CN105704098A - Data transmission method for virtualized networks, node controller and data transmission system for virtualized networks - Google Patents
Data transmission method for virtualized networks, node controller and data transmission system for virtualized networks Download PDFInfo
- Publication number
- CN105704098A CN105704098A CN201410696956.XA CN201410696956A CN105704098A CN 105704098 A CN105704098 A CN 105704098A CN 201410696956 A CN201410696956 A CN 201410696956A CN 105704098 A CN105704098 A CN 105704098A
- Authority
- CN
- China
- Prior art keywords
- nodal point
- point controller
- virtual machine
- data message
- controller
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Computer And Data Communications (AREA)
Abstract
Embodiments of the invention disclose a data transmission method for virtualized networks, a node controller and a data transmission system for virtualized networks. The method comprises the following steps: a first node controller reads a data packet from a storage area of a source virtual machine, wherein the first node controller is a local node controller of the source virtual machine; the data packet is parsed to get a destination address, and a second node controller is determined according to the destination address, wherein the second node controller is a local node controller of a destination virtual machine directed by the destination address; and the data packet is written into the storage area address of the destination virtual machine through the second node controller. A data packet transmitting operation is converted into a storage space shifting operation. The operation is quick, and there is no need to additionally occupy the protocol layer processing ability of NCs. Moreover, performance bottlenecks caused when network card equipment becomes an access hotspot are avoided.
Description
Technical field
The present invention relates to communication technical field, particularly to a kind of data transmission method virtualizing network, Node Controller and system。
Background technology
Nonuniform memory access (Non-uniformmemoryaccess, NUMA) is that a kind of memory construction for multiprocessor designs, and memory access time depends on the core position that processor accesses。Under NUMA, processor accesses the velocity ratio non-local memory (memorizer to the shared processor of another processor or memorizer) of the local storage of its own sooner。
The Non Uniform Memory Access that high-speed cache is relevant accesses a kind of type that (CacheCoherentNon-UniformMemoryAccess, CC-NUMA) is NUMA。In CC-NUMA system, distributed memory is connected formation single internal memory, internal storage access between different NUMA territories is by Node Controller (NodeController, NC) realization is acted on behalf of, the concordance that multiple data copy need not be kept by software, it is not required that software realizes the data of operating system and application system and transmits。
Intel Virtualization Technology computer physical resource such as central processing unit, internal memory, storage and network etc. are given abstract, conversion after present in new ways, make user can fictionalize multiple hardware on a machine, the behavior of simulation real hardware such that it is able to simultaneously run multiple independent operating system。The network service solving the virtualized problem of the network equipment, realizing between different virtual machine is needed when installing multiple virtual machine on an equipment。
Current solution can be such that and uses the network card equipment supporting Intel Virtualization Technology to realize input/output (Input/Output, I/O) collaborative share, and network card equipment realizes multiple functional interface, and each interface assignment gives different virtual machines。The program, it is possible to reduce the resource consumption that central authorities process, it is provided that performance preferably。
But, on the equipment of slotting network interface card, data from Node Controller (NodeController, NC) to network interface card, from network interface card to NC, be required for NC carry out extra protocol processes, add the work load of NC。And waste the protocol layer disposal ability of the NC that the equipment with slotting network interface card is directly connected to。Additionally, owing to a network interface card has been invented multiple Microsoft Loopback Adapter, each virtual machine thinks that Microsoft Loopback Adapter is the equipment of oneself, so each virtual machine all can be sent to the Microsoft Loopback Adapter of oneself when sending data, actually can be sent to the equipment of slotting network interface card, therefore there will be the data transmission between all of virtual machine will through network interface card transfer, and the equipment of slotting network interface card is easy for becoming the focus of access, becomes performance bottleneck。Therefore, above scheme not only wastes the protocol layer disposal ability of NC, and the equipment inserting network interface card easily becomes access focus, causes performance bottleneck。
Summary of the invention
Embodiments provide a kind of data transmission method virtualizing network, Node Controller and system, for saving the protocol layer disposal ability of NC, it is to avoid access, because network card equipment becomes, the performance bottleneck that focus causes。
The embodiment of the present invention provides a kind of data transmission method virtualizing network on the one hand, including:
Primary nodal point controller reads data message from the memory area of source virtual machine；Described primary nodal point controller is the Node Controller that described source virtual machine is local；
Described primary nodal point controller resolves described data message and obtains destination address, and determines secondary nodal point controller according to described destination address；Described secondary nodal point controller is the Node Controller of purpose virtual machine this locality that described destination address points to；
Described primary nodal point controller passes through described secondary nodal point controller, and described data message writes the storage area address of described purpose virtual machine。
In conjunction with implementation on the one hand, in the implementation that the first is possible, described primary nodal point controller passes through described secondary nodal point controller, is included by the storage area address that described data message writes described purpose virtual machine:
Described primary nodal point controller sends address request information to described secondary nodal point controller, then receives the storage area address of the described purpose virtual machine that described secondary nodal point controller returns；
Described data message is write described storage area address by described primary nodal point controller。
In conjunction with the first possible implementation on the one hand, in the implementation that the second is possible, described storage area address is direct memory access DMA address；
Described data message is write described storage area address and includes by described primary nodal point controller:
Described primary nodal point controller processes function by buffer consistency, and described data message is write described storage area address。
In conjunction with implementation on the one hand, in the implementation that the third is possible, described method also includes:
Described primary nodal point controller, after write, sends to described secondary nodal point controller and writes complete message, makes described secondary nodal point controller send to described purpose virtual machine and interrupts and inform that described purpose virtual machine have received described data message。
In conjunction with the third possible implementation on the one hand, in the 4th kind of possible implementation, described primary nodal point controller reads data message from the memory area of source virtual machine, including:
The Microsoft Loopback Adapter engine of described primary nodal point controller reads data message from the DMA transmit queue of source virtual machine；
Described primary nodal point controller passes through described secondary nodal point controller, is included by the storage area address that described data message writes described purpose virtual machine:
The Microsoft Loopback Adapter engine of described primary nodal point controller sends a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone and informs that new data message needs write, and receives the memory headroom of the DMA queue of the Microsoft Loopback Adapter engine return of described secondary nodal point controller；
The Microsoft Loopback Adapter engine of described primary nodal point controller sends write request to the remote protocol engine of described secondary nodal point controller, make described remote protocol engine that described write request to be transmitted to the local protocol engine of described secondary nodal point controller, and make the local protocol engine of described secondary nodal point controller that described data message is write described memory headroom；
Described primary nodal point controller, after write, sends to described secondary nodal point controller and writes complete message, makes described secondary nodal point controller send to described purpose virtual machine and interrupts and inform that described purpose virtual machine have received described data message and includes:
The Microsoft Loopback Adapter engine of described primary nodal point controller is after receiving and writing complete message, write complete message to described in the Microsoft Loopback Adapter engine transmission of described secondary nodal point controller, make the Microsoft Loopback Adapter engine of described secondary nodal point controller notify that the central processing unit of described virtual machine receives described data message。
The embodiment of the present invention two aspect provides a kind of Node Controller, uses as primary nodal point controller, including:
Message reads unit, for reading data message from the memory area of source virtual machine；Described primary nodal point controller is the Node Controller that described source virtual machine is local；
Packet parsing unit, is used for resolving described data message and obtains destination address；
Unit determined by controller, for determining secondary nodal point controller according to described destination address；Described secondary nodal point controller is the Node Controller of purpose virtual machine this locality that described destination address points to；
Data transmission unit, for by described secondary nodal point controller, writing the storage area address of described purpose virtual machine by described data message。
In conjunction with the implementation of two aspects, in the implementation that the first is possible, described data transmission unit includes:
Request transmitting unit, for sending address request information to described secondary nodal point controller；
Address receives unit, for receiving the storage area address of the described purpose virtual machine that described secondary nodal point controller returns；
Message writing unit, for writing described storage area address by described data message。
In conjunction with the first possible implementation of two aspects, in the implementation that the second is possible, described storage area address is direct memory access DMA address；
Described message writing unit, for processing function by buffer consistency, writes described storage area address by described data message。
In conjunction with the implementation of two aspects, in the implementation that the third is possible, described Node Controller, also include:
Message sending unit, after writing at described data message, sends to described secondary nodal point controller and writes complete message, makes described secondary nodal point controller send to described purpose virtual machine and interrupts and inform that described purpose virtual machine have received described data message。
In conjunction with the third possible implementation of two aspects, in the 4th kind of possible implementation, described message reads unit, for reading data message from the DMA transmit queue of source virtual machine by the Microsoft Loopback Adapter engine of described primary nodal point controller；
Described data transmission unit includes: transmitting element is write in notice transmitting element, reception address location and request；
For being sent a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone by the Microsoft Loopback Adapter engine of described primary nodal point controller, described notice transmitting element, informs that new data message needs write；
Described reception address location, the memory headroom of the DMA queue that the Microsoft Loopback Adapter engine for receiving described secondary nodal point controller returns；
Transmitting element is write in described request, for sending write request by the Microsoft Loopback Adapter engine of described primary nodal point controller to the remote protocol engine of described secondary nodal point controller, make described remote protocol engine that described write request to be transmitted to the local protocol engine of described secondary nodal point controller, and make the local protocol engine of described secondary nodal point controller that described data message is write described memory headroom；
Described message sending unit includes: success message receives unit and success message transmitting element；
Described success message receives unit, writes complete message for being received by the Microsoft Loopback Adapter engine of described primary nodal point controller；
Described success message transmitting element, for after the Microsoft Loopback Adapter engine of described primary nodal point controller receives and writes complete message, write complete message to described in the Microsoft Loopback Adapter engine transmission of described secondary nodal point controller, make the Microsoft Loopback Adapter engine of described secondary nodal point controller notify that the central processing unit of described virtual machine receives described data message。
The embodiment of the present invention three aspect provides a kind of data transmission system virtualizing network, including: adopt at least three Node Controller that can communicate to connect, and comprise primary nodal point controller and secondary nodal point controller；Described primary nodal point controller is the Node Controller that source virtual machine is local；The Node Controller that for the purpose of described secondary nodal point controller, virtual machine is local；The Node Controller of any one that described primary nodal point controller provides for the embodiment of the present invention。
As can be seen from the above technical solutions, the embodiment of the present invention has the advantage that the Node Controller resolution data message of source virtual machine side, so that it is determined that the Node Controller of purpose virtual machine side, and obtain the address of memory space, thus realizing the direct memory space that data message is write from source virtual machine purpose virtual machine, achieve and move operation by what the transmission operation of data message became memory space, therefore not only speed is fast, and do not need additionally to take the protocol layer disposal ability of NC, and can avoid accessing, because network card equipment becomes, the performance bottleneck that focus causes。
Accompanying drawing explanation
In order to be illustrated more clearly that the technical scheme in the embodiment of the present invention, below the accompanying drawing used required during embodiment is described is briefly introduced, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the premise not paying creative work, it is also possible to obtain other accompanying drawing according to these accompanying drawings。
Fig. 1 is embodiment of the present invention method flow schematic diagram；
Fig. 2 is embodiment of the present invention method flow schematic diagram；
Fig. 3 is the internal structure schematic diagram of embodiment of the present invention Node Controller；
Fig. 4 is the embodiment of the present invention node internal structure data flow schematic diagram in conjunction with controller；
Fig. 5 is embodiment of the present invention system structure schematic diagram；
Fig. 6 is embodiment of the present invention method flow schematic diagram；
Fig. 7 is embodiment of the present invention method flow schematic diagram；
Fig. 8 is embodiment of the present invention Node Controller structural representation；
Fig. 9 is embodiment of the present invention Node Controller structural representation；
Figure 10 is embodiment of the present invention Node Controller structural representation；
Figure 11 is embodiment of the present invention Node Controller structural representation；
Figure 12 is embodiment of the present invention system structure schematic diagram；
Figure 13 is embodiment of the present invention Node Controller structural representation。
Detailed description of the invention
In order to make the object, technical solutions and advantages of the present invention clearly, below in conjunction with accompanying drawing, the present invention is described in further detail, it is clear that described embodiment is only some embodiments of the present invention, rather than whole embodiments。Based on the embodiment in the present invention, all other embodiments that those of ordinary skill in the art obtain under not making creative work premise, broadly fall into the scope of protection of the invention。
Embodiments provide a kind of data transmission method virtualizing network, as it is shown in figure 1, include:
101: primary nodal point controller reads data message from the memory area of source virtual machine；Above-mentioned primary nodal point controller is the Node Controller that above-mentioned source virtual machine is local；
102: above-mentioned primary nodal point controller resolves above-mentioned data message and obtains destination address, and determines secondary nodal point controller according to above-mentioned purpose address；Above-mentioned secondary nodal point controller is the Node Controller of purpose virtual machine this locality that above-mentioned purpose address is pointed to；
103: above-mentioned primary nodal point controller passes through above-mentioned secondary nodal point controller, above-mentioned data message is write the storage area address of above-mentioned purpose virtual machine。
The embodiment of the present invention, the Node Controller resolution data message of source virtual machine side, so that it is determined that the Node Controller of purpose virtual machine side, and obtain the address of memory space, thus realizing the direct memory space that data message is write from source virtual machine purpose virtual machine, achieve and move operation by what the transmission operation of data message became memory space, therefore not only speed is fast, and do not need additionally to take the protocol layer disposal ability of NC, and can avoid accessing, because network card equipment becomes, the performance bottleneck that focus causes。
More specifically, the present embodiment additionally provides the specific implementation of write data message, as follows: above-mentioned primary nodal point controller passes through above-mentioned secondary nodal point controller, the storage area address that above-mentioned data message writes above-mentioned purpose virtual machine is included:
Above-mentioned primary nodal point controller sends address request information to above-mentioned secondary nodal point controller, then receives the storage area address of the above-mentioned purpose virtual machine that above-mentioned secondary nodal point controller returns；
Above-mentioned data message is write above-mentioned storage area address by above-mentioned primary nodal point controller。
What adopt based on the embodiment of the present invention is the modes directly moved of the data of memory space, more specifically: above-mentioned storage area address is direct memory access DMA address；
Above-mentioned data message is write above-mentioned storage area address and includes by above-mentioned primary nodal point controller:
Above-mentioned primary nodal point controller processes function by buffer consistency, and above-mentioned data message is write above-mentioned storage area address。
What adopt based on the embodiment of the present invention is the modes directly moved of the data of memory space, after data message is written to the memory space of purpose virtual machine side, also needing to inform that purpose virtual machine has been written into complete, concrete scheme is as follows: further, and said method also includes:
Above-mentioned primary nodal point controller, after write, sends to above-mentioned secondary nodal point controller and writes complete message, makes above-mentioned secondary nodal point controller send to above-mentioned purpose virtual machine and interrupts and inform that above-mentioned purpose virtual machine have received above-mentioned data message。
The embodiment of the present invention gives and carries out, based on buffer consistency process function, the specific implementation that internal memory is moved, specific as follows: above-mentioned primary nodal point controller reads data message from the memory area of source virtual machine, including:
The Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller reads data message from the DMA transmit queue of source virtual machine；
Above-mentioned primary nodal point controller passes through above-mentioned secondary nodal point controller, is included by the storage area address that above-mentioned data message writes above-mentioned purpose virtual machine:
The Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller sends a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone and informs that new data message needs write, and receives the memory headroom of the DMA queue of the Microsoft Loopback Adapter engine return of above-mentioned secondary nodal point controller；
The Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller sends write request to the remote protocol engine of above-mentioned secondary nodal point controller, make above-mentioned remote protocol engine that above-mentioned write request to be transmitted to the local protocol engine of above-mentioned secondary nodal point controller, and make the local protocol engine of above-mentioned secondary nodal point controller that above-mentioned data message is write above-mentioned memory headroom；
Above-mentioned primary nodal point controller, after write, sends to above-mentioned secondary nodal point controller and writes complete message, makes above-mentioned secondary nodal point controller send to above-mentioned purpose virtual machine and interrupts and inform that above-mentioned purpose virtual machine have received above-mentioned data message and includes:
The Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller is after receiving and writing complete message, write complete message to the Microsoft Loopback Adapter engine transmission of above-mentioned secondary nodal point controller is above-mentioned, make the Microsoft Loopback Adapter engine of above-mentioned secondary nodal point controller notify that the central processing unit of above-mentioned virtual machine receives above-mentioned data message。
Following example will be described in detail with regard to critical process step, the internal structure of Node Controller and three aspects of data message ablation process of connecting inner controller respectively。In the citing of following example, primary nodal point controller is called source NC, NC for the purpose of secondary nodal point controller, and memory space is DMA region of memory。
One, critical process step:
As in figure 2 it is shown, include:
201: source NC reads data message from the DMA region of memory of source virtual machine；
202: source NC parses purpose Internet protocol (InternetProtocol, IP) address or purpose media interviews control (MediaAccessControl, MAC) address from data message；
203: source NC looks into routing table according to above-mentioned purpose IP address or target MAC (Media Access Control) address, it is determined that purpose NC；
204: source NC to purpose NC sends request, obtains the DMA memory address of the virtual machine of purpose NC；
205: data message is written in the DMA internal memory of purpose virtual machine by source NC；
In above step 205, after data message has write, purpose NC is notified that source NC writes DMA and completes。
206: source NC notice purpose NC writes DMA and completes；
207: purpose NC notice purpose virtual machine CPU receives a new message。
Above flow process can preferably realize flow process as one, purpose NC after omitting step 205 notifies that source NC writes DMA and completes and step 206, directly perform 207 in theory it is also possible that, so NC is carried out bigger amendment by amendment, purpose virtual machine can obtain packet faster, and therefore data transmission bauds can be faster。
Two, the internal structure of Node Controller
As it is shown on figure 3, Node Controller includes such as lower module:
Network interconnection (NetworkInterface) module；
System interface (systeminterface) module；
Protocol schedule controller (Protocoldispatchcontroller, PDC) module；
Remote protocol engine (Remoteprotocolengine, RPE) module；
Local protocol engine (Localprotocalengine, LPE) module；
Microsoft Loopback Adapter engine (VirtualNetworkInterfaceControllerEngine, VNICEngine)；
Wherein Microsoft Loopback Adapter engine comprises again: direct memory access engine (DirectMemoryAccessEngine, DMAEngine), direct memory access management (DirectMemoryAccessEngine, DMAManager) and routing engine (RouteEngine), DMAManager manages DMA context (DMAcontext)。
In Fig. 3, the function of each several part module is as follows:
RPE module acts on behalf of the internal memory of far-end, and the central processing unit (CentralProcessingUnit, CPU) in this end node accesses the internal memory of distant-end node first can send request to RPE；
The local internal memory of LPE module agency, so the memory request that far-end sends will first be dealt into LPE, LPE and initiate to access to local internal memory again；
PDC module is for judging that the type of message is then assigned to LPE, RPE process。
It is also added with a VNICEngine module in the inside of NC, is responsible for processing the affairs that network interface card is relevant。VNICEngine comprises:
DMAEngine, that is responsible for DMA moves work；
DMAManager, is responsible for safeguarding the status information of all of DMA queue and deciding whether to initiate DMA read-write operation；In the present embodiment, virtual machine can one or more DMA queue corresponding, on different NC, DMAManager can also exchange dma information, and handling process below will be described in detail；
RouteEngine, the route being responsible for message judges。
Three, the data message ablation process of connecting inner controller:
In the present embodiment, being Ethernet message transmission path between two virtual machines, as shown in Figure 4, wherein that the left side is the source NC that source virtual machine is corresponding, and the right is the purpose NC that purpose virtual machine is corresponding。The present embodiment is by means of buffer memory (cache) consistency treatment function, and data message can directly be write the DMA internal memory of destination by source NC from the DMA internal memory of source。
The detailed process of data transmission is as follows:
401: CPU is needed the data message sent to put in DMA transmit queue by the driver program of source；I.e. source (source, src) transmit queue；
402: the Microsoft Loopback Adapter engine (NetworkInformationCenterEngine, VNICengine) of source reads data message from source DMA transmit queue；
403: the VNICengine of source notifies that destination VNICengine needs to send new message, and destination VNICengine returns one piece of memory headroom receiving DMA queue；
404: the VNICengine of source initiates to write internal memory operation, and message is written to purpose DMA space；Write memory request and arrive remote protocol engine (Remoteprotocolengine, RPE) module；
Above-mentioned memory request of writing is forwarded to the local protocol engine (LocalProtocolEngine, LPE) of purpose NC by 405:RPE；
The LPE of 406: purpose NC receive RPE write memory request after, data message is written in the receiving queue of purpose virtual machine, namely in write purpose receiving queue (destination, dst), and notifies that source node write operation completes；
The VNICEngine of 407: source NC notifies that the VNICEngine data message of purpose NC is sent completely；
The VNICEngine of 408: purpose NC notifies that the CPU of purpose virtual machine receives a new data message；
409: the CPU of purpose virtual machine takes out message from DMA receiving queue。
Referring to shown in Fig. 5, wherein VM1 is source virtual machine, CPU, the VM1 that CPU2 is corresponding with VM1 have data message be sent to VM2, VM2 corresponding be CPU7；In the network, also include: CPU0, CPU3；NC1 is source NC, NC3 is target NC, also includes NC0 and NC2。
NC internal structure in Fig. 5 is referred to shown in Fig. 3, and VM1 sends datagram and is referred to Fig. 2 or Fig. 4 to the flow process of VM2。
If with reference to flow process shown in Fig. 2, realize in process in " 205: data message is written in the DMA internal memory of purpose virtual machine by source NC ", be specifically referred to scheme implemented as described below:
The process that data message is write the DMA internal memory of purpose virtual machine by source NC is similar with the NC process acting on behalf of CPU write remote memory。According to driver, the mode that DMA internal memory uses can be divided into two kinds of scenes, as follows:
Scene one:
If the DMA internal memory concordance of software assurance data, write internal storage data process as shown in Figure 6, including:
The VNICengine of 601:NC0 sends to the RPE of NC0 and writes memory request；
The RPE of 602:NC0 writes memory request to the LPE forwarding of NC3 is above-mentioned；
The LPE of 603:NC3 has returned the RPE responding to NC0 after being finished；
The RPE of 604:NC0 has forwarded the VNICengine responding to NC0。
Scene two:
If employing hardware to ensure data consistency, before writing internal memory, need first to obtain exclusive authority, idiographic flow as it is shown in fig. 7, comprises:
The VNICengine of 701:NC0 sends the exclusive authority request of request purpose memory address to the RPE of NC0；
The RPE of 702:NC0 forwards the exclusive authority request of above-mentioned request purpose memory address to the LPE of NC3；
The LPE of 703:NC3 determines the exclusive RPE returning later and confirming to respond to NC0；
The RPE of 704:NC0 forwards the VNICengine confirming to respond to NC0；
The VNICengine of 705:NC0 sends to the RPE of NC0 and writes memory request；
The RPE of 706:NC0 writes memory request to the LPE forwarding of NC3 is above-mentioned；
The LPE of 707:NC3 has returned the RPE responding to NC0 after being finished；
The RPE of 708:NC0 has forwarded the VNICengine responding to NC0。
Above example, on data message transmission flow, by means of NC itself and can ensure that the overall situation conforming function of cache, simplify the process of data message transmission。Between two conventional network card equipments during transmission data, after source network interface card reads data from DMA internal memory, forming Ethernet message and issue destination, packet parsing is opened after receiving Ethernet message and is write in DMA internal memory again by destination network interface card。
Owing to NC ensure that the concordance of whole ccNUMA system overall situation cache, so data message directly can be write in the DMA internal memory of destination by the VNICEngine of source by NC, eliminate the functional module that Ethernet message group bag is unpacked, also been removed the reception buffer memory of Ethernet message and stream control, only need the consistency protocol of multiplexing NC to process function。Therefore, it can reduce cpu resource consumption, and NC protocol handling capability is additionally consumed by elimination, thus promoting overall system performance。
The embodiment of the present invention additionally provides a kind of Node Controller, uses as primary nodal point controller, as shown in Figure 8, and including:
Message reads unit 801, for reading data message from the memory area of source virtual machine；Above-mentioned primary nodal point controller is the Node Controller that above-mentioned source virtual machine is local；
Packet parsing unit 802, is used for resolving above-mentioned data message and obtains destination address；
Unit 803 determined by controller, for determining secondary nodal point controller according to above-mentioned purpose address；Above-mentioned secondary nodal point controller is the Node Controller of purpose virtual machine this locality that above-mentioned purpose address is pointed to；
Data transmission unit 804, for by above-mentioned secondary nodal point controller, writing the storage area address of above-mentioned purpose virtual machine by above-mentioned data message。
The embodiment of the present invention, the Node Controller resolution data message of source virtual machine side, so that it is determined that the Node Controller of purpose virtual machine side, and obtain the address of memory space, thus realizing the direct memory space that data message is write from source virtual machine purpose virtual machine, achieve and move operation by what the transmission operation of data message became memory space, therefore not only speed is fast, and do not need additionally to take the protocol layer disposal ability of NC, and can avoid accessing, because network card equipment becomes, the performance bottleneck that focus causes。
More specifically, the present embodiment additionally provides the specific implementation of write data message, as follows: as it is shown in figure 9, above-mentioned data transmission unit 804 includes:
Request transmitting unit 901, for sending address request information to above-mentioned secondary nodal point controller；
Address receives unit 902, for receiving the storage area address of the above-mentioned purpose virtual machine that above-mentioned secondary nodal point controller returns；
Message writing unit 903, for writing above-mentioned storage area address by above-mentioned data message。
What adopt based on the embodiment of the present invention is the modes directly moved of the data of memory space, more specifically: above-mentioned storage area address is direct memory access DMA address；
Above-mentioned message writing unit 903, for processing function by buffer consistency, writes above-mentioned storage area address by above-mentioned data message。
What adopt based on the embodiment of the present invention is the modes directly moved of the data of memory space, after data message is written to the memory space of purpose virtual machine side, also need to inform that purpose virtual machine has been written into complete, concrete scheme is as follows: further, as shown in Figure 10, above-mentioned Node Controller also includes:
Message sending unit 1001, after writing at above-mentioned data message, send to above-mentioned secondary nodal point controller and write complete message, make above-mentioned secondary nodal point controller send to above-mentioned purpose virtual machine and interrupt and inform that above-mentioned purpose virtual machine have received above-mentioned data message。
The embodiment of the present invention gives and carries out, based on buffer consistency process function, the specific implementation that internal memory is moved, specific as follows: alternatively, as shown in figure 11, above-mentioned message reads unit 801, for reading data message from the DMA transmit queue of source virtual machine by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller；
Above-mentioned data transmission unit 804 includes: transmitting element 1103 is write in notice transmitting element 1101, reception address location 1102 and request；
For being sent a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller, above-mentioned notice transmitting element 1101, informs that new data message needs write；
Above-mentioned reception address location 1102, the memory headroom of the DMA queue that the Microsoft Loopback Adapter engine for receiving above-mentioned secondary nodal point controller returns；
Transmitting element 1103 is write in above-mentioned request, for sending write request by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller to the remote protocol engine of above-mentioned secondary nodal point controller, make above-mentioned remote protocol engine that above-mentioned write request to be transmitted to the local protocol engine of above-mentioned secondary nodal point controller, and make the local protocol engine of above-mentioned secondary nodal point controller that above-mentioned data message is write above-mentioned memory headroom；
Above-mentioned message sending unit 1001 includes: success message receives unit 1104 and success message transmitting element 1105；
Above-mentioned success message receives unit 1104, writes complete message for being received by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller；
Above-mentioned success message transmitting element 1105, for after the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller receives and writes complete message, write complete message to the Microsoft Loopback Adapter engine transmission of above-mentioned secondary nodal point controller is above-mentioned, make the Microsoft Loopback Adapter engine of above-mentioned secondary nodal point controller notify that the central processing unit of above-mentioned virtual machine receives above-mentioned data message。
The embodiment of the present invention additionally provides a kind of data transmission system virtualizing network, as shown in figure 12, including: adopt at least three Node Controller 1201 that can communicate to connect, and comprise primary nodal point controller 12011 and secondary nodal point controller 12022；Above-mentioned primary nodal point controller 12011 is the Node Controller that source virtual machine is local；The Node Controller that for the purpose of above-mentioned secondary nodal point controller 12022, virtual machine is local；Wherein, the Node Controller of any one that above-mentioned primary nodal point controller 12011 provides for the embodiment of the present invention。
N NC shown in fig. 12, the NC number of reality is not limited in so, and its number nor affects on the realization of the embodiment of the present invention, and therefore the signal of Figure 12 should not be construed as the uniqueness to the embodiment of the present invention and limits。
The embodiment of the present invention, the Node Controller resolution data message of source virtual machine side, so that it is determined that the Node Controller of purpose virtual machine side, and obtain the address of memory space, thus realizing the direct memory space that data message is write from source virtual machine purpose virtual machine, achieve and move operation by what the transmission operation of data message became memory space, therefore not only speed is fast, and do not need additionally to take the protocol layer disposal ability of NC, and can avoid accessing, because network card equipment becomes, the performance bottleneck that focus causes。
The embodiment of the present invention additionally provides another Node Controller, uses as primary nodal point controller, as shown in figure 13, including: receptor 1301, emitter 1302, processor 1303 and memorizer 1304；
Wherein, processor 1303, for reading data message from the memory area of source virtual machine；Above-mentioned primary nodal point controller is the Node Controller that above-mentioned source virtual machine is local；Resolve above-mentioned data message and obtain destination address, and determine secondary nodal point controller according to above-mentioned purpose address；Above-mentioned secondary nodal point controller is the Node Controller of purpose virtual machine this locality that above-mentioned purpose address is pointed to；By above-mentioned secondary nodal point controller, above-mentioned data message is write the storage area address of above-mentioned purpose virtual machine。
The embodiment of the present invention, the Node Controller resolution data message of source virtual machine side, so that it is determined that the Node Controller of purpose virtual machine side, and obtain the address of memory space, thus realizing the direct memory space that data message is write from source virtual machine purpose virtual machine, achieve and move operation by what the transmission operation of data message became memory space, therefore not only speed is fast, and do not need additionally to take the protocol layer disposal ability of NC, and can avoid accessing, because network card equipment becomes, the performance bottleneck that focus causes。
More specifically, the present embodiment additionally provides the specific implementation of write data message, as follows: above-mentioned processor 1303, for by above-mentioned secondary nodal point controller, the storage area address that above-mentioned data message writes above-mentioned purpose virtual machine is included: send address request information to above-mentioned secondary nodal point controller, then receive the storage area address of the above-mentioned purpose virtual machine that above-mentioned secondary nodal point controller returns；Above-mentioned data message is write above-mentioned storage area address。
What adopt based on the embodiment of the present invention is the modes directly moved of the data of memory space, more specifically: above-mentioned storage area address is direct memory access DMA address；
Above-mentioned processor 1303, includes for above-mentioned data message is write above-mentioned storage area address:
Process function by buffer consistency, above-mentioned data message is write above-mentioned storage area address。
What adopt based on the embodiment of the present invention is the modes directly moved of the data of memory space, after data message is written to the memory space of purpose virtual machine side, also need to inform that purpose virtual machine has been written into complete, concrete scheme is as follows: further, above-mentioned processor 1303, it is additionally operable to after write, send to above-mentioned secondary nodal point controller and write complete message, make above-mentioned secondary nodal point controller send to above-mentioned purpose virtual machine and interrupt and inform that above-mentioned purpose virtual machine have received above-mentioned data message。
The embodiment of the present invention gives and carries out, based on buffer consistency process function, the specific implementation that internal memory is moved, specific as follows: above-mentioned processor 1303, for reading data message from the memory area of source virtual machine, including:
From the DMA transmit queue of source virtual machine, data message is read by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller；
Above-mentioned processor 1303, for by above-mentioned secondary nodal point controller, the storage area address that above-mentioned data message writes above-mentioned purpose virtual machine being included:
Sent a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller and inform that new data message needs write, and the memory headroom of DMA queue that the Microsoft Loopback Adapter engine receiving above-mentioned secondary nodal point controller returns；
Write request is sent to the remote protocol engine of above-mentioned secondary nodal point controller by the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller, make above-mentioned remote protocol engine that above-mentioned write request to be transmitted to the local protocol engine of above-mentioned secondary nodal point controller, and make the local protocol engine of above-mentioned secondary nodal point controller that above-mentioned data message is write above-mentioned memory headroom；
Above-mentioned processor 1303, for, after write, sending to above-mentioned secondary nodal point controller and write complete message, makes above-mentioned secondary nodal point controller send to above-mentioned purpose virtual machine and interrupts and inform that above-mentioned purpose virtual machine have received above-mentioned data message and includes:
By the Microsoft Loopback Adapter engine of above-mentioned primary nodal point controller after receiving and writing complete message, write complete message to the Microsoft Loopback Adapter engine transmission of above-mentioned secondary nodal point controller is above-mentioned, make the Microsoft Loopback Adapter engine of above-mentioned secondary nodal point controller notify that the central processing unit of above-mentioned virtual machine receives above-mentioned data message。
Those skilled in the art is it can be understood that arrive, for convenience and simplicity of description, the system of foregoing description, the specific works process of device and unit, it is possible to reference to the corresponding process in preceding method embodiment, do not repeat them here。
In several embodiments provided herein, it should be understood that disclosed system, apparatus and method, it is possible to realize by another way。Such as, device embodiment described above is merely schematic, such as, the division of described unit, being only a kind of logic function to divide, actual can have other dividing mode when realizing, for instance multiple unit or assembly can in conjunction with or be desirably integrated into another system, or some features can ignore, or do not perform。Another point, shown or discussed coupling each other or direct-coupling or communication connection can be through INDIRECT COUPLING or the communication connection of some interfaces, device or unit, it is possible to be electrical, machinery or other form。
The described unit illustrated as separating component can be or may not be physically separate, and the parts shown as unit can be or may not be physical location, namely may be located at a place, or can also be distributed on multiple NE。Some or all of unit therein can be selected according to the actual needs to realize the purpose of the present embodiment scheme。
It addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, it is also possible to be that unit is individually physically present, it is also possible to two or more unit are integrated in a unit。Above-mentioned integrated unit both can adopt the form of hardware to realize, it would however also be possible to employ the form of SFU software functional unit realizes。
If described integrated unit is using the form realization of SFU software functional unit and as independent production marketing or use, it is possible to be stored in a computer read/write memory medium。Based on such understanding, part or all or part of of this technical scheme that prior art is contributed by technical scheme substantially in other words can embody with the form of software product, this computer software product is stored in a storage medium, including some instructions with so that a computer equipment (can be personal computer, server, or the network equipment etc.) perform all or part of step of method described in each embodiment of the present invention。And aforesaid storage medium includes: USB flash disk, portable hard drive, read only memory (ROM, Read-OnlyMemory), the various media that can store program code such as random access memory (RAM, RandomAccessMemory), magnetic disc or CD。
The above, above example only in order to technical scheme to be described, is not intended to limit；Although the present invention being described in detail with reference to previous embodiment, it will be understood by those within the art that: the technical scheme described in foregoing embodiments still can be modified by it, or wherein portion of techniques feature is carried out equivalent replacement；And these amendments or replacement, do not make the essence of appropriate technical solution depart from the spirit and scope of various embodiments of the present invention technical scheme。
Claims (11)
1. the data transmission method virtualizing network, it is characterised in that including:
Primary nodal point controller reads data message from the memory area of source virtual machine；Described primary nodal point controller is the Node Controller that described source virtual machine is local；
Described primary nodal point controller resolves described data message and obtains destination address, and determines secondary nodal point controller according to described destination address；Described secondary nodal point controller is the Node Controller of purpose virtual machine this locality that described destination address points to；
Described primary nodal point controller passes through described secondary nodal point controller, and described data message writes the storage area address of described purpose virtual machine。
2. method according to claim 1, it is characterised in that described primary nodal point controller passes through described secondary nodal point controller, includes the storage area address that described data message writes described purpose virtual machine:
Described primary nodal point controller sends address request information to described secondary nodal point controller, then receives the storage area address of the described purpose virtual machine that described secondary nodal point controller returns；
Described data message is write described storage area address by described primary nodal point controller。
3. method according to claim 2, it is characterised in that described storage area address is direct memory access DMA address；
Described data message is write described storage area address and includes by described primary nodal point controller:
Described primary nodal point controller processes function by buffer consistency, and described data message is write described storage area address。
4. method according to claim 1, it is characterised in that also include:
Described primary nodal point controller, after write, sends to described secondary nodal point controller and writes complete message, makes described secondary nodal point controller send to described purpose virtual machine and interrupts and inform that described purpose virtual machine have received described data message。
5. method according to claim 4, it is characterised in that described primary nodal point controller reads data message from the memory area of source virtual machine, including:
The Microsoft Loopback Adapter engine of described primary nodal point controller reads data message from the DMA transmit queue of source virtual machine；
Described primary nodal point controller passes through described secondary nodal point controller, is included by the storage area address that described data message writes described purpose virtual machine:
The Microsoft Loopback Adapter engine of described primary nodal point controller sends a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone and informs that new data message needs write, and receives the memory headroom of the DMA queue of the Microsoft Loopback Adapter engine return of described secondary nodal point controller；
The Microsoft Loopback Adapter engine of described primary nodal point controller sends write request to the remote protocol engine of described secondary nodal point controller, make described remote protocol engine that described write request to be transmitted to the local protocol engine of described secondary nodal point controller, and make the local protocol engine of described secondary nodal point controller that described data message is write described memory headroom；
Described primary nodal point controller, after write, sends to described secondary nodal point controller and writes complete message, makes described secondary nodal point controller send to described purpose virtual machine and interrupts and inform that described purpose virtual machine have received described data message and includes:
The Microsoft Loopback Adapter engine of described primary nodal point controller is after receiving and writing complete message, write complete message to described in the Microsoft Loopback Adapter engine transmission of described secondary nodal point controller, make the Microsoft Loopback Adapter engine of described secondary nodal point controller notify that the central processing unit of described virtual machine receives described data message。
6. a Node Controller, uses as primary nodal point controller, it is characterised in that including:
Message reads unit, for reading data message from the memory area of source virtual machine；Described primary nodal point controller is the Node Controller that described source virtual machine is local；
Packet parsing unit, is used for resolving described data message and obtains destination address；
Unit determined by controller, for determining secondary nodal point controller according to described destination address；Described secondary nodal point controller is the Node Controller of purpose virtual machine this locality that described destination address points to；
Data transmission unit, for by described secondary nodal point controller, writing the storage area address of described purpose virtual machine by described data message。
7. Node Controller according to claim 6, it is characterised in that described data transmission unit includes:
Request transmitting unit, for sending address request information to described secondary nodal point controller；
Address receives unit, for receiving the storage area address of the described purpose virtual machine that described secondary nodal point controller returns；
Message writing unit, for writing described storage area address by described data message。
8. Node Controller according to claim 7, it is characterised in that described storage area address is direct memory access DMA address；
Described message writing unit, for processing function by buffer consistency, writes described storage area address by described data message。
9. Node Controller according to claim 6, it is characterised in that also include:
Message sending unit, after writing at described data message, sends to described secondary nodal point controller and writes complete message, makes described secondary nodal point controller send to described purpose virtual machine and interrupts and inform that described purpose virtual machine have received described data message。
10. Node Controller according to claim 9, it is characterised in that
Described message reads unit, for reading data message from the DMA transmit queue of source virtual machine by the Microsoft Loopback Adapter engine of described primary nodal point controller；
Described data transmission unit includes: transmitting element is write in notice transmitting element, reception address location and request；
For being sent a notification message to the Microsoft Loopback Adapter engine of secondary nodal point control zone by the Microsoft Loopback Adapter engine of described primary nodal point controller, described notice transmitting element, informs that new data message needs write；
Described reception address location, the memory headroom of the DMA queue that the Microsoft Loopback Adapter engine for receiving described secondary nodal point controller returns；
Transmitting element is write in described request, for sending write request by the Microsoft Loopback Adapter engine of described primary nodal point controller to the remote protocol engine of described secondary nodal point controller, make described remote protocol engine that described write request to be transmitted to the local protocol engine of described secondary nodal point controller, and make the local protocol engine of described secondary nodal point controller that described data message is write described memory headroom；
Described message sending unit includes: success message receives unit and success message transmitting element；
Described success message receives unit, writes complete message for being received by the Microsoft Loopback Adapter engine of described primary nodal point controller；
Described success message transmitting element, for after the Microsoft Loopback Adapter engine of described primary nodal point controller receives and writes complete message, write complete message to described in the Microsoft Loopback Adapter engine transmission of described secondary nodal point controller, make the Microsoft Loopback Adapter engine of described secondary nodal point controller notify that the central processing unit of described virtual machine receives described data message。
11. virtualize a data transmission system for network, including: adopt at least three Node Controller that can communicate to connect, and comprise primary nodal point controller and secondary nodal point controller；Described primary nodal point controller is the Node Controller that source virtual machine is local；The Node Controller that for the purpose of described secondary nodal point controller, virtual machine is local；It is characterized in that, described primary nodal point controller is Node Controller described in claim 6～10 any one。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410696956.XA CN105704098B (en) | 2014-11-26 | 2014-11-26 | A kind of data transmission method virtualizing network, Node Controller and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410696956.XA CN105704098B (en) | 2014-11-26 | 2014-11-26 | A kind of data transmission method virtualizing network, Node Controller and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105704098A true CN105704098A (en) | 2016-06-22 |
| CN105704098B CN105704098B (en) | 2019-03-01 |
Family
ID=56295253
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410696956.XA Active CN105704098B (en) | 2014-11-26 | 2014-11-26 | A kind of data transmission method virtualizing network, Node Controller and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105704098B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113014631A (en) * | 2021-02-19 | 2021-06-22 | 浙江曲速科技有限公司 | Device cache pushing system and method based on Hlink |
| CN113709139A (en) * | 2021-08-26 | 2021-11-26 | 江苏省未来网络创新研究院 | Openstack east-west forwarding performance optimization method and system based on NUMA architecture |
| CN114880256A (en) * | 2017-04-14 | 2022-08-09 | 华为技术有限公司 | Data processing method, storage system and switching equipment |
| CN115396250A (en) * | 2021-05-09 | 2022-11-25 | 迈络思科技有限公司 | Multi-socket network interface controller with consistent transaction ordering |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009120787A2 (en) * | 2008-03-28 | 2009-10-01 | Mips Technologies, Inc. | Mechanism for maintaining consistency of data written by io devices |
| US7756943B1 (en) * | 2006-01-26 | 2010-07-13 | Symantec Operating Corporation | Efficient data transfer between computers in a virtual NUMA system using RDMA |
| CN103384551A (en) * | 2013-01-04 | 2013-11-06 | 华为技术有限公司 | Virtual machine communication method and system based on PCIE network, and server |
| CN103814596A (en) * | 2013-12-06 | 2014-05-21 | 华为技术有限公司 | Method, device and system for data transmission |
-
2014
- 2014-11-26 CN CN201410696956.XA patent/CN105704098B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7756943B1 (en) * | 2006-01-26 | 2010-07-13 | Symantec Operating Corporation | Efficient data transfer between computers in a virtual NUMA system using RDMA |
| WO2009120787A2 (en) * | 2008-03-28 | 2009-10-01 | Mips Technologies, Inc. | Mechanism for maintaining consistency of data written by io devices |
| CN103384551A (en) * | 2013-01-04 | 2013-11-06 | 华为技术有限公司 | Virtual machine communication method and system based on PCIE network, and server |
| CN103814596A (en) * | 2013-12-06 | 2014-05-21 | 华为技术有限公司 | Method, device and system for data transmission |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114880256A (en) * | 2017-04-14 | 2022-08-09 | 华为技术有限公司 | Data processing method, storage system and switching equipment |
| CN113014631A (en) * | 2021-02-19 | 2021-06-22 | 浙江曲速科技有限公司 | Device cache pushing system and method based on Hlink |
| CN115396250A (en) * | 2021-05-09 | 2022-11-25 | 迈络思科技有限公司 | Multi-socket network interface controller with consistent transaction ordering |
| CN113709139A (en) * | 2021-08-26 | 2021-11-26 | 江苏省未来网络创新研究院 | Openstack east-west forwarding performance optimization method and system based on NUMA architecture |
| CN113709139B (en) * | 2021-08-26 | 2023-03-24 | 江苏省未来网络创新研究院 | Openstack east-west forwarding performance optimization method and system based on NUMA architecture |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105704098B (en) | 2019-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR100241218B1 (en) | Systems and methods for controlling the transmission of relatively large data objects in a communications system | |
| US10698717B2 (en) | Accelerator virtualization method and apparatus, and centralized resource manager | |
| JP5078366B2 (en) | Method, apparatus and program for routing shared I / O fabric error messages to a master control route node in a multi-host environment | |
| CN103703729B (en) | Minimize the virtual machine (vm) migration of packet loss in virtual network | |
| US7356818B2 (en) | Virtual machine communicating to external device without going through other virtual machines by using a list of IP addresses managed only by a single virtual machine monitor | |
| KR102074468B1 (en) | A computer cluster arragement for processing a computation task and method for operation thereof | |
| CN108270676B (en) | Network data processing method and device based on Intel DPDK | |
| US8204054B2 (en) | System having a plurality of nodes connected in multi-dimensional matrix, method of controlling system and apparatus | |
| CN104094230A (en) | System and method for supporting live migration of virtual machines in virtualization environment | |
| TW457437B (en) | Interconnected processing nodes configurable as at least one non-uniform memory access (NUMA) data processing system | |
| CN101635679B (en) | Dynamic update of route table | |
| JP2007086963A (en) | Computer system and i/o bridge | |
| KR19980070065A (en) | System and method for managing the processing of relatively large data objects in a communication stack | |
| JP2010237737A (en) | Apparatus for dynamically migrating lpar with pass-through i/o device, method and program thereof | |
| KR20160087706A (en) | Apparatus and method for resource allocation of a distributed data processing system considering virtualization platform | |
| CN105704098A (en) | Data transmission method for virtualized networks, node controller and data transmission system for virtualized networks | |
| CN108063813B (en) | Method and system for parallelizing password service network in cluster environment | |
| WO2014183531A1 (en) | Method and device for allocating remote memory | |
| US11403250B2 (en) | Operation accelerator, switch, task scheduling method, and processing system | |
| CN102316043A (en) | Port virtualization method, switch and communication system | |
| CN105359122A (en) | Enhanced data transfer in multi-CPU systems | |
| CN112099728B (en) | Method and device for executing write operation and read operation | |
| CN103763403A (en) | Message flow control method, message flow control related device and computational node | |
| CN104571934A (en) | Memory access method, equipment and system | |
| CN112433826B (en) | Hybrid heterogeneous virtualization communication method and chip |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200420 Address after: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee after: HUAWEI TECHNOLOGIES Co.,Ltd. Address before: 301, A building, room 3, building 301, foreshore Road, No. 310052, Binjiang District, Zhejiang, Hangzhou Patentee before: Huawei Technologies Co.,Ltd. |