CN117573602B - Method and computer device for remote direct memory access message transmission - Google Patents
Method and computer device for remote direct memory access message transmission Download PDFInfo
- Publication number
- CN117573602B CN117573602B CN202410059515.2A CN202410059515A CN117573602B CN 117573602 B CN117573602 B CN 117573602B CN 202410059515 A CN202410059515 A CN 202410059515A CN 117573602 B CN117573602 B CN 117573602B
- Authority
- CN
- China
- Prior art keywords
- memory access
- direct memory
- remote direct
- work queue
- queue element
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000005540 biological transmission Effects 0.000 title abstract description 14
- 238000003860 storage Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 14
- 230000002093 peripheral effect Effects 0.000 claims description 8
- 230000000903 blocking effect Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 description 22
- 230000006870 function Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000001427 coherent effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17306—Intercommunication techniques
- G06F15/17331—Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The application relates to the technical field of computers and provides a method for sending a remote direct memory access message and computer equipment. The method comprises the following steps: selectively issuing a first work queue element associated with a first message and a first load corresponding to the first work queue element according to a first issuing mode or a second issuing mode through remote direct memory access software; and acquiring the first work queue element and the first load through remote direct memory access hardware, and then assembling the first work queue element and the first load to obtain a first remote direct memory access packet corresponding to the first message. Therefore, the interface time delay is effectively reduced, and the high-speed data transmission requirement is favorably met.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and a computer device for sending a remote direct memory access packet.
Background
The remote direct memory access is to directly copy the data from the physical line to the application program or directly copy the data from the application program to the physical line, so that the data transmission is performed between two hosts without passing through a system kernel, and the data copying and kernel switching are saved. In the prior art, hardware for sending remote direct memory access data, such as a remote direct memory access network card, is generally connected to a motherboard of a local host through a slot, and the local host realizes interaction with the hardware through bus operation, so that operation delay is high, and requirements of high-speed data transmission are difficult to meet.
Therefore, the application provides a method and computer equipment for sending a remote direct memory access message, which are used for solving the technical problems in the prior art.
Disclosure of Invention
In a first aspect, the present application provides a method for remote direct memory access messaging. The method comprises the following steps: selectively issuing a first work queue element associated with a first message and a first load corresponding to the first work queue element according to a first issuing mode or a second issuing mode through remote direct memory access software; and when the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode, the remote direct memory access software stores the first work queue element and the first load in a first cache space which is not in the remote direct memory access hardware, and when the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode, the remote direct memory access software stores the first work queue element and the first load in a second cache space which is in the remote direct memory access hardware, and when an interface of a first interface which connects the remote direct memory access software and the remote direct memory access hardware is larger than a first preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode.
According to the first aspect of the application, by comparing the interface delay of the first interface connecting the remote direct memory access software and the remote direct memory access hardware with the first preset threshold, the remote direct memory access hardware can be utilized to provide interrupt feedback of the remote direct memory access logic, and the remote direct memory access software selectively issues the first work queue element and the corresponding first load according to the first issuing mode or the second issuing mode, so that the interface delay is effectively reduced, the requirement of high-speed data transmission is favorably met, and meanwhile, the limited calculation force resources and the limited storage resources on the remote direct memory access hardware are also considered.
In a possible implementation manner of the first aspect of the present application, the remote direct memory access software is deployed on a first host, the remote direct memory access hardware is connected to the first host through the first interface, the first cache space is a memory of the first host, and the second cache space is a base address register space of the remote direct memory access hardware.
In a possible implementation manner of the first aspect of the present application, the first interface is a peripheral component interconnect express interface, a universal serial bus interface or a serializer-deserializer interface.
In a possible implementation manner of the first aspect of the present application, when the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode, the remote direct memory access hardware accesses the first buffer space through the first interface to obtain the first work queue element and accesses the first buffer space through the first interface to obtain the first load; when the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access hardware obtains the first work queue element and the first load with a single access to the second cache space and without passing through the first interface.
In a possible implementation manner of the first aspect of the present application, when the remote direct memory access software issues the first work queue element and the first load according to the second issue mode, the remote direct memory access software stores the first work queue element in the second cache space and then appends the first load to a tail of the first work queue element located in the second cache space.
In a possible implementation manner of the first aspect of the present application, when the remote direct memory access software issues the first work queue element and the first load according to the first issue mode, the remote direct memory access hardware receives a doorbell notification from the remote direct memory access software through the first interface; when the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access hardware receives a hardware doorbell notification and does not pass through the first interface.
In a possible implementation manner of the first aspect of the present application, the method further includes: and issuing a plurality of work queue elements and a plurality of loads corresponding to the plurality of work queue elements one by one according to the second issuing mode through the remote direct memory access software, wherein the remote direct memory access software stores the plurality of work queue elements and the plurality of loads in the second cache space through a write combination operation.
In a possible implementation manner of the first aspect of the present application, when the remote direct memory access hardware detects that the link blocking degree is higher than a second preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode, and when the remote direct memory access hardware detects that the base address register resource occupancy rate of the remote direct memory access hardware is higher than a third preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode.
In a possible implementation manner of the first aspect of the present application, the remote direct memory access software is configured to: the first work queue element and the first load are selectively issued in either the first issue mode or the second issue mode based on the number of work queue elements and the data length of the load issued from a previous plurality of batches.
In a possible implementation manner of the first aspect of the present application, when a respective variation amplitude of a number of work queue elements and a data length of a load issued by the previous plurality of batches is smaller than a fourth preset threshold value, the remote direct memory access software issues the first work queue element and the first load according to the second issue mode.
In a possible implementation manner of the first aspect of the present application, when the remote direct memory access software issues the first work queue element and the first load according to the second issue mode, a ratio of a length of the first work queue element to a physical page in the second cache space for storing the first work queue element and the first load is determined based on the first preset threshold.
In a possible implementation manner of the first aspect of the present application, when the application scenario associated with the first message is selected as an industrial automation application scenario or a traffic monitoring application scenario, the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode.
In a second aspect, embodiments of the present application further provide a computer device, the computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing a method according to any one of the implementations of any one of the above aspects when the computer program is executed.
In a third aspect, embodiments of the present application also provide a computer-readable storage medium storing computer instructions that, when run on a computer device, cause the computer device to perform a method according to any one of the implementations of any one of the above aspects.
In a fourth aspect, embodiments of the present application also provide a computer program product comprising instructions stored on a computer-readable storage medium, which when run on a computer device, cause the computer device to perform a method according to any one of the implementations of any one of the above aspects.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for remote direct memory access messaging according to an embodiment of the present application;
FIG. 2 is a schematic diagram of remote direct memory access message sending according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a computing device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
It should be understood that in the description of the application, "at least one" means one or more than one, and "a plurality" means two or more than two. In addition, the words "first," "second," and the like, unless otherwise indicated, are used solely for the purposes of description and are not to be construed as indicating or implying a relative importance or order.
Fig. 1 is a flowchart of a method for sending a remote direct memory access message according to an embodiment of the present application. As shown in fig. 1, the method includes the following steps.
Step S110: and selectively issuing a first work queue element associated with the first message and a first load corresponding to the first work queue element according to a first issuing mode or a second issuing mode through remote direct memory access software.
Step S120: and acquiring the first work queue element and the first load through remote direct memory access hardware, and then assembling the first work queue element and the first load to obtain a first remote direct memory access packet corresponding to the first message.
And when the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode, the remote direct memory access software stores the first work queue element and the first load in a first cache space which is not on the remote direct memory access hardware. When the remote direct memory access software issues the first work queue element and the first load according to the second issue mode, the remote direct memory access software stores the first work queue element and the first load in a second cache space on the remote direct memory access hardware. When the interface delay of a first interface connecting the remote direct memory access software and the remote direct memory access hardware is greater than a first preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode.
Referring to fig. 1, the remote direct memory access software may be an application deployed on the first host for data transfer based on a remote direct memory access technique, including receiving network data from the remote host via a remote direct memory access network, and sending the network data to the remote host via the remote direct memory access network. The remote direct memory access hardware is dedicated hardware connected with the first host and is used for cooperating with the remote direct memory access software to realize data transmission between the first host and the remote host through a remote direct memory access network. Remote direct memory access software may be considered a driver side including drivers for driving hardware and applications for managing data transfers based on remote direct memory access technology. Remote direct memory access hardware can be considered a logical side for providing logical operation functions such as message forwarding processing. Typically, remote direct memory access hardware is connected to the first host through a specific interface, such as through a first interface. The remote direct memory access hardware may be any suitable hardware, such as a network adapter, smart network card, data processor, etc. The remote direct memory access hardware may be deployed on a slot of a motherboard of the first host, interacting with remote direct memory access software on the first host through a bus of the motherboard. The work queue element is a task that the software issues to the hardware and can be used to instruct the hardware where to read how long the data is to be sent to which destination. Here, the first message is a message to be sent out through the remote direct memory access hardware, that is, network data sent to the remote host through the remote direct memory access network. The first work queue element associated with the first message represents a task for sending the first message to be issued by the remote direct memory access software to the remote direct memory access hardware. The first work queue element contains address information and length information, the address information and the length information can be obtained through analyzing the first work queue element, and further, a first load corresponding to the first work queue element is obtained based on the address information and the length information. The first payload represents a data component of a first message to be transmitted. And encapsulating the first load and the message header into a message packet according to a proper message encapsulation format to obtain a first remote direct memory access message packet which is used for being sent to a remote host through a remote direct memory access network.
With continued reference to fig. 1, in step S110, a first work queue element associated with a first message and a first load corresponding to the first work queue element are selectively issued in a first issue mode or a second issue mode by remote direct memory access software; then, in step S120, the first work queue element and the first load are obtained through remote direct memory access hardware, and then the first work queue element and the first load are assembled to obtain a first remote direct memory access packet corresponding to the first packet. Here, the first issue mode means that the remote direct memory access software stores the first work queue element and the first load in a first cache space that is not on the remote direct memory access hardware, and thus, the remote direct memory access hardware obtains the first work queue element and the first load by accessing the first cache space. The second issue mode means that the remote direct memory access software stores the first work queue element and the first load in a second cache space on the remote direct memory access hardware, and therefore the remote direct memory access hardware obtains the first work queue element and the first load by accessing the second cache space. The first cache space is not on the remote direct memory access hardware and the second cache space is on the remote direct memory access hardware. Thus, the remote direct memory access hardware must access the first cache space by means of remote direct memory access software, and thus must involve interaction with the remote direct memory access software, including at least one upstream and downstream operation. The first interface connects the remote direct memory access software and the remote direct memory access hardware, so that the remote direct memory access hardware must access the first cache space to interact with the remote direct memory access software only through the first interface, thereby accessing the first cache space which is not on the remote direct memory access hardware, and thus causing interface delay of the first interface. In contrast, the remote direct memory access hardware needs to access the second cache space, and the second cache space on the remote direct memory access hardware can be directly accessed without using remote direct memory access software, so that interface delay of the first interface is avoided. Considering that the storage resources on the remote direct memory access hardware are limited and that the storage resources on the remote direct memory access hardware are also used for other tasks such as hardware offloading, flow table matching, message forwarding processing, port monitoring, etc., limited storage resources are typically provided to support the storage of work queue elements and loads in the second cache space on the remote direct memory access hardware. Also, the message-related payload portion, e.g., the first message-related first payload, may have various data lengths, depending on the particular communication standard, network protocol, message encapsulation format, etc. For example, in the application scenario of video conferencing, usually one packet contains a record of a conference, and the duration of the conference may range from tens of seconds to tens of minutes, which may cause the data length of the load of the relevant conference packet to change within a larger range of values. Therefore, it is necessary to utilize the limited memory resources on the remote direct memory access hardware to reduce the interface latency, and at the same time, consider factors such as load variation, so as to avoid the excessively high negative effects on the limited computational power resources and the limited memory resources on the remote direct memory access hardware. Therefore, the first work queue element and the first load are selectively issued according to the first issuing mode or the second issuing mode through the remote direct memory access software, and flexible strategy design is facilitated to achieve the balanced optimization effect of the overall system efficiency and the interface delay. Here, when an interface latency of a first interface connecting the remote direct memory access software and the remote direct memory access hardware is greater than a first preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the second issue mode. In this way, by setting the first preset threshold, when the interface delay of the first interface exceeds the first preset threshold, that is, when the interface delay of the first interface is too high, the first work queue element and the first load are issued according to the second issuing mode, which means that the remote direct memory access hardware directly accesses the second cache space on the remote direct memory access hardware, and the remote direct memory access software is not needed, so that the interface delay is reduced. And the interface time delay of accessing the first cache space by the remote direct memory access hardware through the first interface can be recorded every time, so that when the interface time delay exceeds a first preset threshold value, an interrupt can be reported to the remote direct memory access software, and the remote direct memory access software can issue subsequent work queue elements and corresponding loads according to a second issuing mode, thereby reducing the interface time delay. Therefore, by reasonably setting the first preset threshold, the remote direct memory access hardware can be utilized to provide interrupt feedback of the remote direct memory access logic, so that the interface delay is effectively reduced, the high-speed data transmission requirement is favorably met, and meanwhile, the limited computing power resource and the limited storage resource on the remote direct memory access hardware are also considered.
In summary, the method for sending a remote direct memory access message shown in fig. 1 can provide interrupt feedback of remote direct memory access logic by using remote direct memory access hardware by comparing interface delay of a first interface connecting remote direct memory access software and remote direct memory access hardware with a first preset threshold, and selectively issue a first work queue element and a corresponding first load according to a first issue mode or a second issue mode by using the remote direct memory access software, thereby effectively reducing interface delay, being beneficial to meeting high-speed data transmission requirements, and also considering limited computing power resources and limited storage resources on the remote direct memory access hardware.
In one possible implementation manner, the remote direct memory access software is deployed on a first host, the remote direct memory access hardware is connected to the first host through the first interface, the first cache space is a memory of the first host, and the second cache space is a base address register space of the remote direct memory access hardware. In some embodiments, the first interface is a peripheral component interconnect express interface, a universal serial bus interface, or a serializer-deserializer interface. In this way, by comparing the interface delay of the first interface connecting the remote direct memory access software and the remote direct memory access hardware with the first preset threshold, the remote direct memory access hardware can be utilized to provide interrupt feedback of the remote direct memory access logic, and the remote direct memory access software selectively issues the first work queue element and the corresponding first load according to the first issuing mode or the second issuing mode, thereby effectively reducing the interface delay, being beneficial to meeting the requirement of high-speed data transmission, and simultaneously taking the limited computing power resource and the limited storage resource on the remote direct memory access hardware into consideration. For example, when the first interface is a shortcut peripheral component interconnect interface, a shortcut peripheral component interconnect interface delay caused by each shortcut peripheral component interconnect interface operation may be compared with a first predetermined threshold. The first preset threshold may be set to 20 microseconds, such that interrupt feedback of the remote direct memory access logic is triggered when the PCI express delay exceeds 20 microseconds. In addition, the second cache space is a base address register space of the remote direct memory access hardware, which facilitates fully utilizing storage resources of the base address register space.
In a possible implementation manner, when the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode, the remote direct memory access hardware accesses the first cache space through the first interface to acquire the first work queue element and accesses the first cache space through the first interface to acquire the first load; when the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access hardware obtains the first work queue element and the first load with a single access to the second cache space and without passing through the first interface. In this way, when the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode, the remote direct memory access hardware needs to perform two consecutive interface operations of the first interface to obtain the first work queue element and the first load, thereby resulting in two interface delays. When the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode, the remote direct memory access hardware can acquire the first work queue element and the first load only by single access operation and does not cause interface delay. Further, in some embodiments, when the remote direct memory access software issues the first work queue element and the first payload in the second issue mode, the remote direct memory access software stores the first work queue element in the second cache space and then appends the first payload to a tail of the first work queue element located in the second cache space. In this way, the remote direct memory access software adds the first load to the tail of the first work queue element, so that the subsequent remote direct memory access hardware can acquire the first work queue element and the first load through a single access, thereby being beneficial to reducing the data processing delay. Further, in some embodiments, when the remote direct memory access software issues the first work queue element and the first load in the first issue mode, the remote direct memory access hardware receives a doorbell notification from the remote direct memory access software through the first interface; when the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access hardware receives a hardware doorbell notification and does not pass through the first interface. Therefore, the hardware doorbell notification speed is faster than that of the remote direct memory access software, and the hardware doorbell notification does not need to pass through the first interface, so that the data processing delay is reduced.
In one possible embodiment, the method further comprises: and issuing a plurality of work queue elements and a plurality of loads corresponding to the plurality of work queue elements one by one according to the second issuing mode through the remote direct memory access software, wherein the remote direct memory access software stores the plurality of work queue elements and the plurality of loads in the second cache space through a write combination operation. Thus, by writing the combination operation, the data can be issued in batches, for example, 64 work queue elements and corresponding loads can be issued at a time, or 8 work queue elements and corresponding loads can be issued at a time. The method can be flexibly adjusted by combining the available storage resources capable of supporting the second cache space, generally, when the number of work queue elements issued in batches is large, the load information carried by the work queue elements is small, and the method is suitable for input-output intensive applications, for example, work queue elements with the size of 64 bytes are uniformly used. When the number of work queue elements issued in batches is small, the load information carried by the work queue elements is large, for example, work queue elements with the size of 512 bytes are uniformly used.
In one possible implementation, the remote direct memory access software issues the first work queue element and the first load according to the second issue mode when the remote direct memory access hardware detects that a link blocking level is above a second preset threshold, and issues the first work queue element and the first load according to the first issue mode when the remote direct memory access hardware detects that a base address register resource occupancy of the remote direct memory access hardware is above a third preset threshold. The remote direct memory access hardware may periodically detect the link blocking degree, and evaluate the link blocking state, for example, when the network delay is relatively large, and for example, when the remote direct memory access packet needs to be frequently retransmitted, which may mean network congestion, and may trigger interrupt feedback of the remote direct memory access logic. In addition, when the remote direct memory access hardware finds that the occupancy rate of the base address register resource is high, in order to avoid affecting the normal operation of other functions such as hardware offloading and the like, the first work queue element and the first load may be interrupted or switched to be issued according to the first issue mode. For example, a third preset threshold value may be set to 90%, i.e. when the base address register resource occupancy exceeds 90%, the first work queue element and the first load are interrupted or switched to be issued in the first issue mode. Therefore, on the basis of comparing the interface delay of the first interface connecting the remote direct memory access software and the remote direct memory access hardware with the first preset threshold, the detection of the link blocking degree and the second preset threshold can be further utilized, and a more comprehensive control mechanism is established by utilizing the occupancy rate of the base address register resource and the third preset threshold to control the remote direct memory access software to issue the work queue element and the corresponding load according to the first issuing mode or the second issuing mode, so that the interface delay is effectively reduced, the requirement of high-speed data transmission is favorably met, and meanwhile, the limited calculation power resource and the limited storage resource on the remote direct memory access hardware are also considered.
In one possible implementation, the remote direct memory access software is configured to: the first work queue element and the first load are selectively issued in either the first issue mode or the second issue mode based on the number of work queue elements and the data length of the load issued from a previous plurality of batches. In this way, the remote direct memory access software, that is, the remote direct memory access driver, adopts a batch-by-batch issuing manner, and can determine whether the message data to be sent has regularity, that is, changes within a certain constraint range, by comparing the number of work queue elements and the data length of the load of the previous batches, such as the previous batch and the next batch. Thus, by using the remote direct memory access software, on the basis of comparing the interface delay of the first interface connecting the remote direct memory access software and the remote direct memory access hardware with the first preset threshold, a more comprehensive control mechanism can be established, and the constraint can be realized: only if the number of work queue elements and the data length of the load issued from the previous batches vary by a certain extent, for example not more than 10% or 15%, will the work queue elements and the corresponding loads continue to be issued in the second issue mode. In this way, considering the limited computational power resources and limited storage resources on the remote direct memory access hardware, when the number of work queue elements and the data length of the load issued by the batch change more severely, for example, the number exceeds the set constraint range, that is, the regularity is not provided, a larger processing load is possibly brought to the remote direct memory access hardware, so that the overall negative influence exceeds the positive benefit. In some embodiments, the remote direct memory access software issues the first work queue element and the first load in the second issue mode when the respective magnitudes of changes in the number of work queue elements and the data length of the load issued by the previous plurality of batches are less than a fourth preset threshold. Therefore, on the basis of comparing the interface time delay of the first interface connecting the remote direct memory access software and the remote direct memory access hardware with the first preset threshold, a more comprehensive control mechanism is established to control the remote direct memory access software to issue the work queue elements and the corresponding loads according to the first issuing mode or the second issuing mode by utilizing the respective variation amplitude of the number of the work queue elements and the data length of the loads and the fourth preset threshold, so that the interface time delay is effectively reduced, the requirement of high-speed data transmission is favorably met, and meanwhile, the limited computing power resources and the limited storage resources on the remote direct memory access hardware are also considered.
In one possible implementation, when the remote direct memory access software issues the first work queue element and the first payload in the second issue mode, a ratio of a length of the first work queue element relative to a physical page in the second cache space for storing the first work queue element and the first payload is determined based on the first preset threshold. Here, the first preset threshold represents a latency requirement, and by comparing an interface latency of a first interface connecting the remote direct memory access software and the remote direct memory access hardware with the first preset threshold, a control mechanism can be established based on the latency requirement to control the remote direct memory access software to issue the work queue element and the corresponding load according to the first issue mode or the second issue mode. Here, the ratio of the length of the first work queue element to the physical page, e.g. the ratio of 64 bytes size work queue element to 4 kilobytes of physical page is about 64 times smaller, which ratio is adjustable in connection with the user requirements and the application scenario requirements. The higher the ratio is, the fewer work queue elements are contained in the same physical page, the load information is increased, and thus the processing delay is possibly increased; the lower the ratio setting, the more work queue elements that are held in the same physical page, the less load information that is carried, which may result in reduced processing latency. Therefore, in a scenario with high latency requirement, for example, a low latency application scenario with a fast response speed, a lower proportion may be set, that is, the proportion of the length of the first work queue element relative to the physical pages used for storing the first work queue element and the first load in the second cache space may be determined based on the first preset threshold, so that the user requirement and the application scenario requirement may be better adapted.
In one possible implementation, when the application scenario associated with the first message is selected as an industrial automation application scenario or a traffic monitoring application scenario, the remote direct memory access software issues the first work queue element and the first load according to the second issue mode. In this way, the remote direct memory access software may be controlled to issue the work queue element and the corresponding load according to the first issue mode or the second issue mode by user intervention or other modes according to the requirements of the application scenario. Industrial automation or traffic monitoring applications generally mean that the number of work queue elements and the data length of the load change relatively regularly, in which applications the data sender typically sends a recorded video data stream with a fixed duration.
Fig. 2 is a schematic diagram of remote direct memory access message transmission according to an embodiment of the present application. As shown in fig. 2, a first work queue element 202 associated with a first message and a first payload 204 corresponding to the first work queue element 202 are selectively issued in a first issue mode or a second issue mode by remote direct memory access software 210. The first work queue element 202 and the first load 204 are obtained by the remote direct memory access hardware 220, and then the first work queue element 202 and the first load 204 are assembled to obtain a first remote direct memory access packet 206 corresponding to the first packet. A first interface 230 connects the remote direct memory access software 210 and the remote direct memory access hardware 220 is shown in fig. 2. Also shown in FIG. 2, when the remote direct memory access software 210 issues the first work queue element 202 and the first payload 204 in the second issue mode, the remote direct memory access software 210 stores the first work queue element 202 and the first payload 204 in a second cache space 222 on the remote direct memory access hardware 220. The remote direct memory access hardware 220 includes a message forwarding processing engine 224, the message forwarding processing engine 224 obtains the first work queue element 202 and the first payload 204 from a second buffer space 222 on the remote direct memory access hardware 220 and sends a first remote direct memory access packet 206 to the outside. The remote direct memory access software 210 also includes an application 212 for data transfer based on remote direct memory access technology, including receiving network data from a remote host over a remote direct memory access network and transmitting the network data to the remote host over the remote direct memory access network.
By comparing the interface delay of the first interface 230 connecting the remote direct memory access software 210 and the remote direct memory access hardware 220 with the first preset threshold, the remote direct memory access hardware 220 can be used to provide interrupt feedback of the remote direct memory access logic, and the remote direct memory access software 210 selectively issues the first work queue element 202 and the corresponding first load 204 according to the first issue mode or the second issue mode, thereby effectively reducing the interface delay, being beneficial to meeting the requirement of high-speed data transmission, and simultaneously considering the limited computing power resources and the limited storage resources on the remote direct memory access hardware.
Fig. 3 is a schematic structural diagram of a computing device according to an embodiment of the present application, where the computing device 300 includes: one or more processors 310, a communication interface 320, and a memory 330. The processor 310, the communication interface 320 and the memory 330 are interconnected by a bus 340. Optionally, the computing device 300 may further include an input/output interface 350, where the input/output interface 350 is connected to an input/output device for receiving parameters set by a user, etc. The computing device 300 can be used to implement some or all of the functionality of the device embodiments or system embodiments of the present application described above; the processor 310 can also be used to implement some or all of the operational steps of the method embodiments described above in connection with the embodiments of the present application. For example, specific implementations of the computing device 300 performing various operations may refer to specific details in the above-described embodiments, such as the processor 310 being configured to perform some or all of the steps of the above-described method embodiments or some or all of the operations of the above-described method embodiments. For another example, in an embodiment of the present application, the computing device 300 may be configured to implement some or all of the functionality of one or more components of the apparatus embodiments described above, and the communication interface 320 may be configured to implement communication functions and the like necessary for the functionality of the apparatus, components, and the processor 310 may be configured to implement processing functions and the like necessary for the functionality of the apparatus, components.
It should be appreciated that the computing device 300 of fig. 3 may include one or more processors 310, and that the plurality of processors 310 may cooperatively provide processing power in a parallelized connection, a serialized connection, a serial-parallel connection, or any connection, or the plurality of processors 310 may constitute a processor sequence or processor array, or the plurality of processors 310 may be separated into primary and secondary processors, or the plurality of processors 310 may have different architectures such as employing heterogeneous computing architectures. In addition, the computing device 300 shown in FIG. 3, the associated structural and functional descriptions are exemplary and not limiting. In some example embodiments, computing device 300 may include more or fewer components than shown in fig. 3, or combine certain components, or split certain components, or have a different arrangement of components.
Processor 310 may take many specific forms, for example, processor 310 may include one or more combinations of a central processing unit (central processing unit, CPU), a graphics processor (graphic processing unit, GPU), a neural network processor (neural-network processing unit, NPU), a tensor processor (tensor processing unit, TPU), or a data processor (data processing unit, DPU), and embodiments of the present application are not limited in this respect. Processor 310 may also be a single-core processor or a multi-core processor. The processor 310 may be formed by a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (programmable logic device, PLD), or a combination thereof. The PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (FPGA) GATE ARRAY, generic array logic (GENERIC ARRAY logic, GAL), or any combination thereof. Processor 310 may also be implemented solely with logic devices incorporating processing logic, such as an FPGA or Digital Signal Processor (DSP), etc. The communication interface 320 may be a wired interface, which may be an ethernet interface, a local area network (local interconnect network, LIN), etc., or a wireless interface, which may be a cellular network interface, or use a wireless lan interface, etc., for communicating with other modules or devices.
The memory 330 may be a nonvolatile memory such as a read-only memory (ROM), a Programmable ROM (PROM), an erasable programmable ROM (erasable PROM, EPROM), an electrically erasable programmable EPROM (EEPROM), or a flash memory. Memory 330 may also be volatile memory, which may be random access memory (random access memory, RAM) used as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (STATIC RAM, SRAM), dynamic random access memory (DYNAMIC RAM, DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (double DATA RATE SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (ENHANCED SDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCHLINK DRAM, SLDRAM), and direct memory bus random access memory (direct rambus RAM, DR RAM). Memory 330 may also be used to store program code and data such that processor 310 invokes the program code stored in memory 330 to perform some or all of the operational steps of the method embodiments described above, or to perform corresponding functions in the apparatus embodiments described above. Moreover, computing device 300 may contain more or fewer components than shown in FIG. 3, or may have a different configuration of components.
Bus 340 may be a peripheral component interconnect express (PERIPHERAL COMPONENT INTERCONNECT EXPRESS, PCIe) bus, or an extended industry standard architecture (extended industry standard architecture, EISA) bus, a unified bus (unified bus, ubus or UB), a computer quick link (compute express link, CXL), a cache coherent interconnect protocol (cache coherent interconnect for accelerators, CCIX), or the like. The bus 340 may be divided into an address bus, a data bus, a control bus, and the like. The bus 340 may include a power bus, a control bus, a status signal bus, and the like in addition to a data bus. But is shown with only one bold line in fig. 3 for clarity of illustration, but does not represent only one bus or one type of bus.
The method and the device provided by the embodiment of the application are based on the same inventive concept, and because the principle of solving the problem by the method and the device is similar, the embodiment, the implementation, the example or the implementation of the method and the device can be mutually referred, and the repetition is not repeated. Embodiments of the present application also provide a system comprising a plurality of computing devices, each of which may be structured as described above. The functions or operations that may be implemented by the system may refer to specific implementation steps in the above method embodiments and/or specific functions described in the above apparatus embodiments, which are not described herein.
Embodiments of the present application also provide a computer-readable storage medium having stored therein computer instructions which, when executed on a computer device (e.g., one or more processors), implement the method steps of the method embodiments described above. The specific implementation of the processor of the computer readable storage medium in executing the above method steps may refer to specific operations described in the above method embodiments and/or specific functions described in the above apparatus embodiments, which are not described herein again.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. The application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Embodiments of the application may be implemented, in whole or in part, in software, hardware, firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein. The computer program product includes one or more computer instructions. When loaded or executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). Computer readable storage media can be any available media that can be accessed by a computer or data storage devices, such as servers, data centers, etc. that contain one or more collections of available media. Usable media may be magnetic media (e.g., floppy disks, hard disks, tape), optical media, or semiconductor media. The semiconductor medium may be a solid state disk, or may be a random access memory, flash memory, read only memory, erasable programmable read only memory, electrically erasable programmable read only memory, register, or any other form of suitable storage medium.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. Each flow and/or block of the flowchart and/or block diagrams, and combinations of flows and/or blocks in the flowchart and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments. It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present application without departing from the spirit or scope of the embodiments of the application. The steps in the method of the embodiment of the application can be sequentially adjusted, combined or deleted according to actual needs; the modules in the system of the embodiment of the application can be divided, combined or deleted according to actual needs. The present application is also intended to include such modifications and alterations if they come within the scope of the claims and the equivalents thereof.
Claims (12)
1. A method for remote direct memory access messaging, the method comprising:
Selectively issuing a first work queue element associated with a first message and a first load corresponding to the first work queue element according to a first issuing mode or a second issuing mode through remote direct memory access software;
acquiring the first work queue element and the first load through remote direct memory access hardware, then assembling the first work queue element and the first load to obtain a first remote direct memory access packet corresponding to the first message,
Wherein when the remote direct memory access software issues the first work queue element and the first load in the first issue mode, the remote direct memory access software stores the first work queue element and the first load in a first cache space that is not on the remote direct memory access hardware,
When the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access software stores the first work queue element and the first load in a second cache space on the remote direct memory access hardware,
When an interface latency of a first interface connecting the remote direct memory access software and the remote direct memory access hardware is greater than a first preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the second issue mode,
The remote direct memory access software side is deployed on a first host, the remote direct memory access hardware is connected with the first host through the first interface, the first cache space is the memory of the first host, the second cache space is the base address register space of the remote direct memory access hardware,
When the remote direct memory access hardware detects that the link blocking degree is higher than a second preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the second issuing mode, and when the remote direct memory access hardware detects that the base address register resource occupancy rate of the remote direct memory access hardware is higher than a third preset threshold, the remote direct memory access software issues the first work queue element and the first load according to the first issuing mode.
2. The method of claim 1, wherein the first interface is a peripheral component interconnect express interface, a universal serial bus interface, or a serializer-deserializer interface.
3. The method of claim 1, wherein when the remote direct memory access software issues the first work queue element and the first load in the first issue mode, the remote direct memory access hardware accesses the first cache space through the first interface to obtain the first work queue element and accesses the first cache space through the first interface to obtain the first load; when the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access hardware obtains the first work queue element and the first load with a single access to the second cache space and without passing through the first interface.
4. The method of claim 3, wherein when the remote direct memory access software issues the first work queue element and the first payload in the second issue mode, the remote direct memory access software stores the first work queue element in the second cache space and then appends the first payload to a tail of the first work queue element located in the second cache space.
5. The method of claim 3, wherein the remote direct memory access hardware receives a doorbell notification from the remote direct memory access software through the first interface when the remote direct memory access software issues the first work queue element and the first load in the first issue mode; when the remote direct memory access software issues the first work queue element and the first load in the second issue mode, the remote direct memory access hardware receives a hardware doorbell notification and does not pass through the first interface.
6. The method according to claim 1, wherein the method further comprises:
And issuing a plurality of work queue elements and a plurality of loads corresponding to the plurality of work queue elements one by one according to the second issuing mode through the remote direct memory access software, wherein the remote direct memory access software stores the plurality of work queue elements and the plurality of loads in the second cache space through a write combination operation.
7. The method of claim 1, wherein the remote direct memory access software is configured to: the first work queue element and the first load are selectively issued in either the first issue mode or the second issue mode based on the number of work queue elements and the data length of the load issued from a previous plurality of batches.
8. The method of claim 7, wherein the remote direct memory access software issues the first work queue element and the first load in the second issue mode when the respective magnitudes of the changes in the number of work queue elements and the data length of the load issued by the previous plurality of batches are less than a fourth preset threshold.
9. The method of claim 1, wherein when the remote direct memory access software issues the first work queue element and the first payload in the second issue mode, a ratio of a length of the first work queue element relative to a physical page in the second cache space for storing the first work queue element and the first payload is determined based on the first preset threshold.
10. The method of claim 1, wherein the remote direct memory access software issues the first work queue element and the first load in the second issue mode when the application scenario associated with the first message is selected as an industrial automation application scenario or a traffic monitoring application scenario.
11. A computer device, characterized in that it comprises a memory, a processor and a computer program stored on the memory and executable on the processor, which processor implements the method according to any of claims 1 to 10 when executing the computer program.
12. A computer readable storage medium storing computer instructions which, when run on a computer device, cause the computer device to perform the method of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410059515.2A CN117573602B (en) | 2024-01-16 | 2024-01-16 | Method and computer device for remote direct memory access message transmission |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410059515.2A CN117573602B (en) | 2024-01-16 | 2024-01-16 | Method and computer device for remote direct memory access message transmission |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117573602A CN117573602A (en) | 2024-02-20 |
| CN117573602B true CN117573602B (en) | 2024-05-14 |
Family
ID=89864754
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410059515.2A Active CN117573602B (en) | 2024-01-16 | 2024-01-16 | Method and computer device for remote direct memory access message transmission |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117573602B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110069431A (en) * | 2018-01-24 | 2019-07-30 | 上海交通大学 | Elastic Key-Value key-value pair data storage method based on RDMA and HTM |
| CN112463654A (en) * | 2019-09-06 | 2021-03-09 | 华为技术有限公司 | Cache implementation method with prediction mechanism |
| CN115695458A (en) * | 2022-10-28 | 2023-02-03 | 青岛民航凯亚系统集成有限公司 | Data storage method of BS (base station) terminal under weak network environment |
| CN115934625A (en) * | 2023-03-15 | 2023-04-07 | 珠海星云智联科技有限公司 | Doorbell knocking method, device and medium for remote direct memory access |
| WO2023098050A1 (en) * | 2021-11-30 | 2023-06-08 | 华为技术有限公司 | Remote data access method and apparatus |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9965441B2 (en) * | 2015-12-10 | 2018-05-08 | Cisco Technology, Inc. | Adaptive coalescing of remote direct memory access acknowledgements based on I/O characteristics |
-
2024
- 2024-01-16 CN CN202410059515.2A patent/CN117573602B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110069431A (en) * | 2018-01-24 | 2019-07-30 | 上海交通大学 | Elastic Key-Value key-value pair data storage method based on RDMA and HTM |
| CN112463654A (en) * | 2019-09-06 | 2021-03-09 | 华为技术有限公司 | Cache implementation method with prediction mechanism |
| WO2023098050A1 (en) * | 2021-11-30 | 2023-06-08 | 华为技术有限公司 | Remote data access method and apparatus |
| CN115695458A (en) * | 2022-10-28 | 2023-02-03 | 青岛民航凯亚系统集成有限公司 | Data storage method of BS (base station) terminal under weak network environment |
| CN115934625A (en) * | 2023-03-15 | 2023-04-07 | 珠海星云智联科技有限公司 | Doorbell knocking method, device and medium for remote direct memory access |
Non-Patent Citations (1)
| Title |
|---|
| 基于嵌入式MPMC接口的内存访问机制;常振超 等;《计算机工程》;20120731;第38卷(第13期);第231-233页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117573602A (en) | 2024-02-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8694595B2 (en) | Low latency, high bandwidth data communications between compute nodes in a parallel computer | |
| US8018951B2 (en) | Pacing a data transfer operation between compute nodes on a parallel computer | |
| US7948999B2 (en) | Signaling completion of a message transfer from an origin compute node to a target compute node | |
| US20080313376A1 (en) | Heuristic Status Polling | |
| US7779173B2 (en) | Direct memory access transfer completion notification | |
| US20100268852A1 (en) | Replenishing Data Descriptors in a DMA Injection FIFO Buffer | |
| US20090031001A1 (en) | Repeating Direct Memory Access Data Transfer Operations for Compute Nodes in a Parallel Computer | |
| US20090031055A1 (en) | Chaining Direct Memory Access Data Transfer Operations for Compute Nodes in a Parallel Computer | |
| CN115934623B (en) | Data processing method, device and medium based on remote direct memory access | |
| CN115934625B (en) | Doorbell knocking method, equipment and medium for remote direct memory access | |
| US20090031002A1 (en) | Self-Pacing Direct Memory Access Data Transfer Operations for Compute Nodes in a Parallel Computer | |
| CN114153778A (en) | Cross-network bridging | |
| US20180181421A1 (en) | Transferring packets between virtual machines via a direct memory access device | |
| US6256660B1 (en) | Method and program product for allowing application programs to avoid unnecessary packet arrival interrupts | |
| EP0871307A2 (en) | Apparatus for flexible control of interrupts in multiprocessor systems | |
| US7890597B2 (en) | Direct memory access transfer completion notification | |
| US10305772B2 (en) | Using a single work item to send multiple messages | |
| CN117573602B (en) | Method and computer device for remote direct memory access message transmission | |
| US20230342086A1 (en) | Data processing apparatus and method, and related device | |
| CN117573603B (en) | Data processing method and computer equipment for remote direct memory access | |
| CN116340246B (en) | Data pre-reading method and medium for direct memory access read operation | |
| WO2023040412A1 (en) | Message processing method, network card and server | |
| US20220179805A1 (en) | Adaptive pipeline selection for accelerating memory copy operations | |
| CN117527654B (en) | Method and system for analyzing network traffic packet | |
| CN117687795B (en) | Hardware offloading method, device and medium for remote direct memory access |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |