WO2014201623A1 - Procédé, appareil et système de transmission de données, et carte de réseau physique - Google Patents

Procédé, appareil et système de transmission de données, et carte de réseau physique Download PDF

Info

Publication number
WO2014201623A1
WO2014201623A1 PCT/CN2013/077412 CN2013077412W WO2014201623A1 WO 2014201623 A1 WO2014201623 A1 WO 2014201623A1 CN 2013077412 W CN2013077412 W CN 2013077412W WO 2014201623 A1 WO2014201623 A1 WO 2014201623A1
Authority
WO
WIPO (PCT)
Prior art keywords
module
module group
group
network card
physical network
Prior art date
Application number
PCT/CN2013/077412
Other languages
English (en)
Chinese (zh)
Inventor
赵晓雷
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN201380000801.1A priority Critical patent/CN103609077B/zh
Priority to PCT/CN2013/077412 priority patent/WO2014201623A1/fr
Publication of WO2014201623A1 publication Critical patent/WO2014201623A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/14—Multichannel or multilink protocols
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Definitions

  • the present invention relates to the field of communications and, more particularly, to a method, apparatus and system for data transmission and a physical network card. Background technique
  • NIC virtualization is the key technology to solve the last mile of virtualization. It has experienced software simulation, Virtual Machine Device Queue (VMDq), and single root I/O virtualization (Single-Root I). /O
  • SR-IOV Virtualization
  • PCIe Peripheral Component Interconnect Express
  • the NIC device is virtualized into at least two PCIe NIC devices, and each virtual PCIe NIC device provides services to the upper layer software of the virtual machine just like the physical PCIe NIC device.
  • SR-IOV is a 10 virtualization technology standard that allows one physical device to support at least two virtual NICs. SR-IOV introduces two new types of functions, Physical Function (Physical Function,
  • the PF module has all the PCIe functions of the SR-IOV, and can perform SR-IOV function configuration and management.
  • the VF module has a lightweight PCIe function, including resources necessary for data transmission and a small amount of configuration resources.
  • the VF module can be managed via Virtual Machine Management (Virtual Machine Management,
  • VMM is assigned to the virtual machine (VM).
  • Embodiments of the present invention provide a method, an apparatus, and a system for data transmission, and a physical network card, which can avoid system failure and reduce resource waste.
  • the first aspect provides a method for data transmission, including: determining, from a physical function PF module group of a first physical NIC, a primary PF module, where the PF module group includes at least two PF modules, and the PF module group Each of the PF modules corresponds to a different virtual function VF module group; based on the VF module group corresponding to each PF module in the PF module group, the target VF module group managed by the main PF module is determined, wherein the VF in the target VF module group
  • the module is configured to be allocated to at least one virtual machine VM such that the at least one VM implements data transmission.
  • determining, by the PF module group of the first physical network card, the primary PF module includes: determining, according to a reference of each PF module in the PF module group a parameter, the primary PF module is determined, wherein the reference parameter of the PF module includes at least one of a PF module failure number, a number of VF modules in the VF module group corresponding to the PF module, a static bandwidth of the PF module, and a static bandwidth of the PF module.
  • the determining, according to the VF module group corresponding to each PF module in the PF module group, determining the primary The target VF module group managed by the PF module includes: selecting at least one VF module from the VF module group corresponding to each PF module in the PF module group to form the target VF module group.
  • the method further includes: determining The first PF module is re-enabled or fault-recovered, and the first PF module is determined as the PF module in the PF module group. Based on the first PF module, the VF module group corresponding to the first PF module is established.
  • the method further includes: determining the second PF module Deleting from the PF module group of the first physical network card; releasing the VF module group corresponding to the second PF module.
  • the method further includes: determining the first physics When the third PF module of the network card fails, determining whether the VF module in the VF module group corresponding to the third PF module is available; when at least one VF module in the VF module group corresponding to the third PF module is available, The at least one VF available in the VF module group corresponding to the first PF module The module is determined to continue to be available.
  • the method further comprises: determining the target
  • the first VF module allocated to the first VM of the at least one VM in the VF module group is faulty; from the VF module group corresponding to each PF module in the PF module group, selecting a second different from the first VF module VF module; update the target VF module group.
  • the first physical network card occupies at least two high speeds
  • the peripheral component interconnects the standard PCIe bus
  • the method further includes: determining that data of the VF module connected to the first PCIe bus of the at least two PCIe buses in the target VF module group needs to be sent to the target VF module group a VF module connected to the second PCIe bus of the at least two PCIe buses; and the data of the VF module connected to the first PCIe bus is sent to the VF module connected to the second PCIe bus through the switch of the first physical network card.
  • the method further includes: determining the primary PF module A fault occurs; from the PF module group, a PF module other than the main PF module is selected as a new main PF module; and the new main PF module is determined based on the VF module group corresponding to each PF module in the PF module group.
  • a new target VF module group is managed, wherein the VF module in the new target VF module group is used to allocate to at least one virtual machine VM, so that the at least one VM implements data transmission.
  • the second aspect provides an apparatus for data transmission, including: a first determining unit, configured to determine, from a physical function PF module group of the first physical network card, a primary PF module, where the PF module group includes at least two PF module, each PF module in the PF module group corresponds to a different virtual function VF module group; and a second determining unit is configured to determine the main PF module management based on the virtual function VF module group corresponding to each PF module in the PF module group
  • the target VF module group wherein the VF module in the target VF module group is used to allocate to at least one virtual machine VM, so that the at least one VM implements data transmission.
  • the first determining unit is specifically configured to: determine, according to a reference parameter of each PF module in the PF module group, the primary PF module, where The reference parameters of the PF module include at least the number of PF module failures, the number of VF modules in the VF module group corresponding to the PF module, the static bandwidth of the PF module, and the static bandwidth of the PF module.
  • the reference parameters of the PF module include at least the number of PF module failures, the number of VF modules in the VF module group corresponding to the PF module, the static bandwidth of the PF module, and the static bandwidth of the PF module.
  • the second determining unit is specifically configured to: use each PF module from the PF module group At least one VF module is selected from the corresponding VF module group to form the target VF module group.
  • the apparatus further includes: a determining unit, configured to determine that the first PF module is re-enabled or fault-recovered, determining the first PF module as a PF module in the PF module group; and establishing a unit, configured to establish the first PF module based on the first PF module A VF module group corresponding to a PF module.
  • the device further includes: a fifth determining unit, The second PF module is configured to be deleted from the PF module group of the first physical NIC; and the release unit is configured to release the VF module group corresponding to the second PF module.
  • the device further includes: a sixth determining unit, When it is determined that the third PF module of the first physical network card is faulty, determining whether the VF in the VF module group corresponding to the third PF module is available; the holding unit, configured to be in the VF module group corresponding to the third PF module When at least one VF module is available, the at least one VF module in the VF module group corresponding to the third PF module is determined to continue to be available.
  • the device further includes: a seventh determining unit, Determining that a first VF module allocated to the first VM of the at least one VM in the target VF module group is faulty; and selecting a unit for selecting from a VF module group corresponding to each PF module in the PF module group a second VF module different from the first VF module; an update unit, configured to update the target VF module group, and assign the second VF module to the first VM.
  • the first physical network card occupies at least two high speeds
  • the peripheral component interconnects the standard PCIe bus
  • the device further includes: an eighth determining unit, configured to determine that data of the VF module connected to the first PCIe bus of the at least two PCIe buses in the target VF module group needs to be sent to a VF module connected to the second PCIe bus of the at least two PCIe buses in the target VF module group; a sending unit, configured to connect the VF to the first PCIe bus The data of the module is sent to the VF module connected to the second PCIe bus through the switch of the first physical network card.
  • the device further includes a third determining unit, Determining that the main PF module is faulty; the first determining unit is further configured to: select, from the PF module group, a PF module other than the main PF module as a new main PF module; the second determining unit further uses And: determining, according to the VF module group corresponding to each PF module in the PF module group, a new target VF module group managed by the new main PF module, where the VF module in the new target VF module group is used to allocate to At least one virtual machine VM, such that the at least one VM implements data transmission.
  • a third determining unit Determining that the main PF module is faulty; the first determining unit is further configured to: select, from the PF module group, a PF module other than the main PF module as a new main PF module; the second determining unit further uses And: determining, according to the VF module group corresponding to each PF module in the PF module group, a new target VF module
  • an apparatus for data transmission including a processor and a memory; wherein the memory stores program code, the processor is configured to call the program code stored in the memory, and performs the following operations: Physical NIC module physical group PF module group, the main PF module is determined, the PF module group includes at least two PF modules, each PF module in the PF module group corresponds to a different virtual function VF module group; based on each of the PF module groups a virtual function VF module group corresponding to the PF module, determining a target VF module group managed by the main PF module, wherein the target
  • the VF module in the VF module group is used to allocate to at least one virtual machine VM, so that the at least one
  • the VM implements data transfer.
  • the processor is configured to invoke the program code stored in the memory, and specifically perform the following operations: based on a reference of each PF module in the PF module group a parameter, the primary PF module is determined, wherein the reference parameters of the PF module include the number of failures of the PF module, the number of VF modules in the VF module group corresponding to the PF module,
  • At least one of a static bandwidth of the PF module and a static bandwidth of the PF module At least one of a static bandwidth of the PF module and a static bandwidth of the PF module.
  • the processor is configured to invoke the program code stored in the memory, and specifically perform the following operations: At least one VF module is selected from the VF module group corresponding to each PF module in the PF module group to form the target VF module group.
  • the processor is used to invoke the The program code stored in the memory further performs the following operations: determining that the first PF module is re-enabled or failback, determining the first PF module as the PF module in the PF module group; establishing based on the first PF module The VF module group corresponding to the first PF module.
  • the processor is configured to invoke the storage in the memory
  • the program code also performs the following operations: determining that the second PF module is deleted from the PF module group of the first physical network card; releasing the VF module group corresponding to the second PF module.
  • the processor is configured to invoke the storage in the memory
  • the program code is further configured to: determine whether the VF module in the VF module group corresponding to the third PF module is available when the third PF module of the first physical network card is faulty; and corresponding to the third PF module When at least one VF module in the VF module group is available, the at least one VF module available in the VF module group corresponding to the first PF module is determined to be continuously available.
  • the processor is configured to invoke the storage in the memory
  • the program code is further configured to: determine that the first VF module allocated to the first VM of the at least one VM in the target VF module group is faulty; from the VF module group corresponding to each PF module in the PF module group Selecting a second VF module different from the first VF module; updating the target VF module group, and assigning the second VF module to the first VM.
  • the first physical network card occupies at least two high speeds
  • the peripheral component interconnects a standard PCIe bus
  • the processor is configured to call the program code stored in the memory, and performs the following operations: determining that the target PCI module is required to be connected to the first PCIe bus of the at least two PCIe buses
  • the data of the VF module is sent to the VF module in the target VF module group and connected to the second PCIe bus of the at least two PCIe buses; and the data of the VF module connected to the first PCIe bus passes through the first physical network card.
  • the switch sends to the VF module connected to the second PCIe bus.
  • the processor is configured to invoke the storage in the memory
  • the program code also performs the following operations: determining that the main PF module is faulty; from the PF module group, selecting a PF module other than the main PF module as a new main PF module; based on each PF in the PF module group a VF module group corresponding to the module, determining a new target VF module group managed by the new main PF module, wherein the VF module in the new target VF module group is used to allocate to at least one virtual machine VM, so that the at least one The VM implements data transfer.
  • a physical network card including: a PF module group having at least two PF modules, and at least two VF module groups; wherein, one of the at least two PF modules corresponds to the at least one One of the two VF module groups, one of the at least two PF modules can manage the VF module in the VF module group to which the other PF modules are connected after being determined as the main PF module.
  • the VF module in each VF module group of the at least two VF module groups includes two types of spatial resources; a type of space resource is used to access the PF module corresponding to the VF module group to which the VF module to which the two types of space resources belong before the primary PF module is determined; the other two of the two types of space resources are accessed.
  • a type of spatial resource is used to establish a connection with the primary PF module for access by the primary PF module after the primary PF module is determined.
  • the physical network card occupies at least two high-speed peripheral components interconnecting a standard PCIe bus
  • the physical network card further includes a built-in switch; wherein the built-in switch is configured to send data of the VF module connected to the first PCIe bus of the at least two PCIe buses to the at least two PCIe buses The VF module connected to the second PCIe bus.
  • the physical network card passes at least two a port is connected to the external switch; the physical network card is further configured to receive a data stream sent by the device for data transmission through the aggregated link, and uniformly send the data stream to the external switch by using each of the at least two ports .
  • a fifth aspect a system for data transmission, comprising the apparatus for data transmission in any of the possible implementations of the second aspect or the second aspect, and the fourth aspect or the fourth aspect
  • the physical network card in the implementation manner wherein the first physical network card or the third aspect or the third aspect of the possible implementation in any one of the possible implementation manners in the second aspect or the second aspect
  • the first physical network card referred to in the manner is the physical network card referred to in the fourth aspect or any possible implementation manner of the fourth aspect.
  • one physical network card has a PF module group composed of at least two PF modules, and one PF can be selected from at least two PF modules in the PF module group.
  • the module acts as the main PF module, which avoids system failures and reduces resource waste due to the many opportunities for selection.
  • FIG. 1 is a schematic flow chart of a method for data transmission according to an embodiment of the present invention.
  • FIG. 2 is a schematic flow chart of a method for data transmission according to another embodiment of the present invention.
  • 3 is a schematic block diagram of an apparatus for data transmission in accordance with another embodiment of the present invention.
  • 4 is a schematic block diagram of an apparatus for data transmission in accordance with another embodiment of the present invention.
  • FIG. 5 is a schematic block diagram of an apparatus for data transmission according to another embodiment of the present invention.
  • 6 is a schematic block diagram of a physical network card in accordance with another embodiment of the present invention.
  • FIG. 7 is a schematic block diagram of a system for data transmission in accordance with another embodiment of the present invention. detailed description
  • the method 100 includes:
  • S110 Determine a primary PF module from a first PF module group of the first physical NIC, where the first PF module group includes at least two PF modules, and each PF module in the first PF module group corresponds to a different VF module group.
  • a device for data transmission for example, a device having at least one processor for data transmission function, such as a host, or a host group composed of at least two hosts, which can be from the first physics Determining a main PF module in a PF module group including at least two PF modules of the network card; and then determining, from the VF module group corresponding to each PF module in the PF module group, a target VF module group that needs to be managed by the main PF module, where The VF module in the target VF module group can then be assigned to at least one virtual machine, so that each virtual machine can implement data transmission through the VF module.
  • one physical network card has at least two PF modules to form a first PF module group, and one PF module can be selected as a main PF module from at least two PF modules in the first PF module group. Since there are at least two PF modules to choose from, the reliability of the main PF module is increased, thereby reducing the probability of system failure and reducing resource waste caused by system failure.
  • the physical network card mentioned in the embodiment of the present invention may also be referred to as an SR-IOV network card, which may include two functional modules, a PF module and a VF module, which those skilled in the art should know, also referred to as PF and VF.
  • the PF module supports the PCIe function of the SR-IOV extended function, which is used to configure and manage the SR-IOV function;
  • the VF module implements the PCIe function.
  • the virtual machine management can treat each VF module in the target VF module group as A PCIe device is assigned to a virtual machine (VM).
  • the VM When the VM performs control plane access to the assigned VF module, the VM initiates a request, the primary PF module and the VMM respond to the request, and return the result to the VM.
  • the VM sends and receives data packets from the VF module, it combines the input/output memory management unit (IMMMU) technology provided by the CPU to realize complete and independent transmission and reception.
  • IMMMU input/output memory management unit
  • each PF module in the PF module group has a corresponding relationship with the VF module group.
  • each PF module can be used to manage all VF modules in the corresponding VF module group. That is to say, the VF modules in the corresponding VF module group are hung under each PF module.
  • the main PF module can implement unified management and resource allocation for the VF modules in the VF module group corresponding to all PF modules in the PF module group, so that the VF modules corresponding to all PF modules can be obtained. All or part of the VF modules in the group are selected to form a target VF module group, and the VF module in the target VF module group is allocated to the virtual machine by the VMM.
  • the VF module may include two types of space resources, and the first type of space resource is controlled by the directly corresponding PF module. Before the primary PF module is determined, the directly corresponding PF module accesses the first type of space resource. Manage the VF module; the second type of space resource is in the system After the primary PF module is selected, it is remapped to the area visible to the primary PF module (that is, the connection between the second spatial resource and the primary PF module is established), and the primary PF module can directly access the second type of spatial resource to implement The VF module is managed and is not controlled by the directly corresponding PF module.
  • determining the primary PF module from the PF module group of the first physical network card in S110 may include: determining the primary PF module based on a reference parameter of each PF module in the PF module group, where The reference parameter may include at least one of a number of failures of the PF module, a number of VF modules in the VF module group corresponding to the PF module, a static bandwidth of the PF module, and a dynamic bandwidth of the PF module.
  • the apparatus for data transmission may determine each of the PF module groups based on the reference parameter of each PF module in the PF module group.
  • the performance value of a PF module, and based on the performance value of each PF module, the priority ordering of the PF modules in the PF module group can be determined, and then the main PF module can be determined from the PF module group based on the priority ordering.
  • the reference parameter of each PF module can be obtained according to different reference performance values with respect to the PF module.
  • the baseline performance value of one failure can be set to -10, and the reference performance value of the VF module is 5, if the number of failures of PF module 0 is 2 and the number of VF modules in the corresponding VF module group is 6, the number of failures of PF module 1 is 3, and the number of VF modules in the corresponding VF module group is 5, then It can be calculated that the performance value of PF module 0 is 10, and the performance value of PF module 1 is -5, then the performance value of PF module 0 is greater than the performance value of PF module 1, and thus, the priority of PF module 0 is higher than that of PF module. 1 , you can select PF module 0 as the main PF module. It should be understood that the foregoing is only an example, and the embodiment of the present invention is not limited thereto.
  • the following principles can be determined for how to determine the primary PF module: If the reference parameter is only the number of PF module failures, the fewer the number of failures occurs, the higher the priority of the PF module is, and the more likely it is to be determined as the primary PF.
  • the reference parameter is only the number of VF modules in the VF module group corresponding to the PF module, the more the number of VF modules in the VF module group corresponding to the PF module, the higher the priority of the PF module is, the more likely it is to be
  • the reference parameter is only the static bandwidth of the PF module, the greater the static bandwidth of the PF module, the higher the priority of the PF module, the more likely it is to be determined as the primary PF module;
  • the reference parameter is only the PF module Dynamic bandwidth, the larger the dynamic bandwidth of the PF module, the higher the priority, the more likely it is to be determined as the primary PF module; if the reference parameter is the number of PF module failures, the number of VF modules in the VF module group corresponding to the PF module PF mode
  • At least two of the block dynamic bandwidth and the PF module static bandwidth can be prioritized by considering two factors to determine the main PF module.
  • the target VF module group managed by the main PF module is determined according to the VF module group corresponding to each PF module in the PF module group, and may include: each PF module from the first PF module group. At least one VF module is selected from the corresponding VF module group to form the target VF module group.
  • VF modules when at least two VF modules are required to implement the link aggregation function (otherwise, in other scenarios), after the primary PF module is determined, it may be from the VF module group corresponding to each PF module in the first PF module group. Select at least one VF module to form the target VF module group.
  • a certain PF module fails, the normal operation of the VF module in the VF module group corresponding to the other PF modules is not affected, so that the VF modules of different VF module groups can be isolated from each other, so that the present invention
  • Embodiments Selecting a VF module from a VF module group corresponding to different PF modules can improve system reliability and stability, thereby ensuring system bandwidth.
  • the aggregated link data stream needs to be sent (the aggregated link data stream refers to the data stream jointly sent by the at least two VF modules, that is, the channel through which at least two VF modules send and receive data can form an aggregated link.
  • the aggregated link data stream can be sent to the external switch through multiple ports on the physical NIC.
  • the traffic of each port can be monitored in real time.
  • the physical NIC has two ports connected to the external switch.
  • the sending scheduling module in the physical network card detects that the traffic of port 0 is 0.2 G, and the traffic of port 1 is 1 G.
  • the aggregation link sent to port 1 can be reduced.
  • the traffic is increased to the traffic of the aggregated link of port 0.
  • the traffic of port 0 and port 1 can be adjusted to 0.6G.
  • the data stream can also be sent to the external switch through multiple ports, and the traffic of the port can be monitored and adjusted in real time.
  • the VF module group corresponding to the main PF module may be directly determined as the target VF module group managed by the main PF module, or one or more other ones or more
  • the VF module group corresponding to the PF module is determined as the target VF module group managed by the main PF module.
  • the target VF module group is determined from the VF module group corresponding to each PF module in the PF module group, and the present invention is not limited.
  • the method 100 may further include: determining that the first PF module is re-enabled or failing, and determining the first PF module as the PF module in the first PF module group; The first PF module establishes a VF module group corresponding to the first PF module.
  • the method 100 may further include: determining that the second PF module is deleted from the first PF module group of the first physical network card; releasing the VF module group corresponding to the second PF module.
  • the means for data transmission can monitor all VF modules of the first physical network card and manage the VF modules in groups.
  • the VF module connected to the PF module 0 can be composed of the VF module group 0, that is, the VF module 0, the VF module 1 and the VF module 127 form the VF module group 0, and the VF module connected to the PF module 1 constitutes the VF module group.
  • 1 that is, the VF module 128, the VF module 129, the VF module 130...
  • the VF module 255 constitutes the VF module group 1.
  • the device for data transmission may establish a VF corresponding to the first PF module based on the first PF module and the VF module connected to the first PF module.
  • Module group When the second PF module in the PF module group of the first physical network card is deleted from the system (for example, the second PF module is deleted or the user actively deletes), the established second PF module may be released. Group of VF modules.
  • the VF module group corresponding to the re-enabled or fault-recovering PF module may be established, or after the PF module is deleted, the deleted is deleted.
  • the VF module group corresponding to the PF module can dynamically adjust the VF module that can be used in the physical network card, making the system more flexible.
  • the method 100 may further include: determining whether the VF module in the VF module group corresponding to the third PF module is available when the third PF module of the first physical network card is faulty; When at least one VF module in the VF module group corresponding to the third PF module is available, the at least one VF available in the VF module group corresponding to the first PF module is determined to be continuously available.
  • the VF module group corresponding to the PF module may not be directly released, and the state of the VF module under the PF module may be further determined. If all the VF modules under the PF module are faulty, the VF module group corresponding to the PF module can be directly released. If a part of the VF module under the PF module fails, only the failed VF module needs to be removed from the VF. Excluded from the module group.
  • the available VF module that is connected to the failed PF module is determined to continue to be available, thereby avoiding resource waste.
  • first PF module the second PF module and the third PF mentioned in the embodiments of the present invention Modules can be the same PF module or different PF modules.
  • the method 100 may further include: determining that a first VF module allocated to the first VM of the at least one VM in the target VF module group is faulty; corresponding to each PF module in the PF module group In the VF module group, a second VF module is determined, the second VF module is different from the VF module included in the target VF module group; the target VF module group is updated, and the second VF module is allocated to the first VM.
  • a new VF module can be reselected from the VF module group corresponding to each PF module in the PF module group of the physical network card, and the reselected VF is selected.
  • the module is different from the VF module included in the original target VF module group; then, the target VF module group is updated, that is, the reselected VF module is used as a member of the target VF module group, and the reselected VF module is assigned to The VM.
  • the method 100 may further include: determining that data of the VF module connected to the first PCIe bus in the target VF module group needs to be sent to the VF connected to the second PCIe bus in the target VF module group. And transmitting, by the switch of the first physical network card, the data of the VF module connected to the first PCIe bus to the VF module connected to the second PCIe bus.
  • the first physical network card can occupy at least two PCIe buses. If one VF module data on one PCIe bus needs to be sent to another VF module on the same PCIe bus, the data can be implemented through an internal switch of the physical network card. Send. If the VF module data connected to one PCIe bus needs to be sent to the VF module connected to another PCIe bus, the data can also be transmitted through the internal switch of the physical network card.
  • data exchange between VF modules connected to different PCIe buses in the same physical network card, or data exchange between VF modules connected to the same PCIe bus in the same physical network card may pass the data exchange.
  • the switch built in the physical NIC can save network bandwidth and reduce data transmission and reception delay.
  • the method 100 may further include: determining that the primary PF module is faulty; and selecting, from the PF module group, a PF module other than the primary PF module as a new primary PF module; a VF module group corresponding to each PF module in the module group, determining a new target VF module group managed by the new main PF module; assigning the VF module in the new target VF module group to at least one virtual machine, so that At least one virtual machine implements data transfer.
  • the specific operation information of the main PF module can be tracked in real time, and the state of the main PF module is monitored. If the main PF module fails, it can be from PF Reselecting a PF module as a new main PF module in the module group, updating the recorded operation information to the new main PF module, and re-determining the new target VF module group for the new main PF module, and The VF module in the new target VF module group is allocated to at least one virtual machine, so that the at least one virtual machine implements data transmission.
  • a new primary PF module when the original primary PF module fails, a new primary PF module can be reselected from the PF module group having at least two PF modules, which can further avoid system failure and reduce resource waste.
  • a second PF module group can also exist in the network card, and the PF module in the second PF module group can virtualize at least two by software simulation.
  • vNICs for use by legacy virtual machines.
  • the PF module group 1 includes the PF module 0; the PF module group 2 includes the PF module 1 and the PF module 2, the PF module 1 has the VF module 1 and the VF module 2 attached thereto, and the PF module 2 has the VF module 3 attached thereto.
  • PF module 0, PF module 1 and PF module 2 are shown as PF0, PF1 and PF2 in the figure; VF module 1, VF module 2, VF module 3 and VF module 4 are respectively shown as VF 1, VF2, VF3 and VF4.
  • the VF DD shown in Figure 2 refers to the drive of the VF module (Device Driver), and the PF DD refers to the drive of the PF module.
  • the VMM can be run by the host for VM creation, deletion, management of hardware resources, allocation of VM resources, and so on. The method according to an embodiment of the present invention will be specifically described below with reference to FIG.
  • the VMM can select a PF module as the main PF module from the PF module group 2 in the physical network card (NIC). Specifically, the VMM can be based on the number of failures of each PF module, the number of VF modules in the corresponding VF module group, and the PF module. At least one of the static bandwidth and the dynamic bandwidth of the PF module determines the primary PF module. For example, as shown in FIG. 2, the PF module 1 may be determined as the primary PF module. After the main PF module is determined, the VMM can determine the VF module to be managed by the main PF module. If the link aggregation function is required, the VF module can be selected from the VF modules connected to different PF modules.
  • NIC physical network card
  • the target VF module group managed by the PF module for example, as shown in FIG. 2, the VF module 2, the VF module 3, and the VF module 4 can be determined as the target VF module group. Then, the VMM can allocate each VF module in the target VF module group that needs to be managed by the main PF module to each VM for subsequent data transmission of each VM, for example, as shown in FIG. 2, The VF module 2 is assigned to the VM 2, the VF module 3 is assigned to the VM 3, and the VF module 4 is assigned to the VM 4.
  • the VMM can reselect a PF module from the PF module group as the main PF module, and can reselect the target VF module group for the main PF module.
  • the VMM may also virtualize one or at least two vNICs through the PF module based on the software simulation manner. For example, as shown in FIG. 2, the vNIC1 and the vNIC 2 are virtualized by the PF module 0, respectively. Legacy) VM1 and legacy (legacy) VM2 services. After the VF module or vNIC is assigned to the VM, each VM can implement data transmission. The following describes the implementation of the four main data transmission paths:
  • Data Stream 1 Data exchange between Legacy VMs is achieved through VMM's built-in virtual software switch (Vi r tua l SW swi tch ).
  • Data Stream 2 The data channels sent and received by the virtual NIC in the Legacy VM are: External Switch Internal Switch - MAC - PF - VMM - VM; VM - VMM - PF - MAC - Internal Switch - External Switch.
  • Data Stream 3 The data exchange between the VMs using the VF module is implemented by the switch built in the NIC. Whether the VF module is connected to the same PCIe or a different PCIe, it can be implemented through the switch built into the NIC.
  • the VM data transceiver channel using the VF module is: External Switch - Internal Switch - Media Access Control (MAC) ⁇ VF; VF - MAC - Internal Switch - External Switch.
  • MAC External Switch - Internal Switch - Media Access Control
  • Data stream 5 The channel through which the primary PF performs resource management and control for the VF is: VM ⁇ VMM - primary PF-VF; VF-primary PF ⁇ VMM ⁇ VM.
  • a physical network card has a PF module group composed of at least two PF modules, and one PF module can be selected as a main PF module from at least two PF modules in the PF module group, thereby Due to the many opportunities for selection, system failures can be avoided and resource waste can be reduced.
  • FIG. 3 is a schematic block diagram of an apparatus 200 for data transmission in accordance with an embodiment of the present invention. As shown
  • the apparatus 200 includes a first determining unit 210 and a second determining unit 220.
  • the first determining unit 210 is configured to determine, from the physical function PF module group of the first physical network card, the main PF module, where the PF module group includes at least two PF modules, and each PF module in the PF module group corresponds to a different VF module group.
  • the second determining unit 220 is configured to determine, according to the virtual function VF module group corresponding to each PF module in the PF module group, the target VF module group managed by the main PF module, where the VF module in the target VF module group is used for Assigned to at least one virtual machine VM so that the at least one VMs implement data transfer.
  • the first determining unit 210 is specifically configured to: determine, according to a reference parameter of each PF module in the PF module group, the primary PF module, where the reference parameter of the PF module includes a PF module failure number, a PF module At least one of the number of corresponding VF modules, the static bandwidth of the PF module, and the static bandwidth of the PF module.
  • the second determining unit 220 is specifically configured to: select at least one VF module from the VF module group corresponding to each PF module in the PF module group to form the target VF module group.
  • the apparatus 200 further includes a fourth determining unit 252 and an establishing unit 254.
  • the fourth determining unit 252 is configured to determine that the first PF module is re-enabled or fault-recovered, and the first PF module is determined as a PF module in the PF module group; and the establishing unit 254 is configured to be based on the first PF module. Establishing a VF module group corresponding to the first PF module.
  • the apparatus 200 further includes a fifth determining unit 256 and a releasing unit 258.
  • the fifth determining unit 256 is configured to determine that the second PF module is deleted from the PF module group of the first physical network card; and the releasing unit 258 is configured to release the VF module group corresponding to the second PF module.
  • the apparatus 200 further includes a sixth determining unit 262, a holding unit 264, and a holding unit 264.
  • the sixth determining unit 262 is configured to determine whether the VF in the VF module group corresponding to the third PF module is available when the third PF module of the first physical network card is faulty; the holding unit 264 is configured to be in the When at least one VF module in the VF module group corresponding to the third PF module is available, the at least one VF module in the VF module group corresponding to the third PF module is determined to continue to be available.
  • the apparatus 200 further includes a seventh determining unit 272, a selecting unit 274, and an updating unit 276.
  • the seventh determining unit 272 is configured to determine that the first VF module allocated to the first VM of the at least one VM in the target VF module group is faulty; the selecting unit 274 is configured to use the VF corresponding to each PF module in the PF module group. In the module group, a second VF module different from the first VF module is selected; the update unit 276 is used to update the target VF module group.
  • the apparatus 200 further includes: an eighth determining unit 282, configured to determine that the target needs to be Transmitting, by the VF module group, the data of the VF module connected to the first PCIe bus of the at least two PCIe buses to the VF module of the target VF module group and the second PCIe bus of the at least two PCIe buses; a sending unit 284, configured to connect to the first PCIe bus The data of the connected VF module is sent to the VF module connected to the second PCIe bus through the switch of the first physical network card.
  • the apparatus 200 further includes a third determining unit 240.
  • the third determining unit 240 is configured to determine that the primary PF module is faulty; the first determining unit 210 is further configured to: select, from the PF module group, a PF module other than the primary PF module as a new primary PF.
  • the second determining unit 220 is further configured to: determine, according to the VF module group corresponding to each PF module in the PF module group, a new target VF module group managed by the new primary PF module, where the new target VF The VF module in the module group is used to allocate to at least one virtual machine VM, so that the at least one VM implements data transmission.
  • a physical network card has a PF module group composed of at least two PF modules, and one PF module can be selected as a main PF module from at least two PF modules in the PF module group, thereby Due to the many opportunities for selection, system failures can be avoided and resource waste can be reduced.
  • FIG. 5 is a schematic block diagram of an apparatus 300 for data transmission in accordance with an embodiment of the present invention.
  • the apparatus 300 includes: a processor 310310 and a memory 320320; wherein the memory 320 stores program code, and the processor 310 is configured to invoke the program code stored in the memory 320 to perform the following operations: The physical function of a physical network card in the PF module group, determine the main
  • the PF module group includes at least two PF modules, and each PF module in the PF module group corresponds to a different virtual function VF module group; and based on the virtual function VF module group corresponding to each PF module in the PF module group, a target VF module group managed by the main PF module, wherein, the target
  • the VF module in the VF module group is used to allocate to at least one virtual machine VM, so that the at least one
  • the VM implements data transfer.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and specifically perform the following operations: determining the primary PF module based on a reference parameter of each PF module in the PF module group, where the PF The reference parameters of the module include at least one of the number of PF module failures, the number of VF modules corresponding to the PF module, the static bandwidth of the PF module, and the static bandwidth of the PF module.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and specifically perform the following operations: selecting at least one VF module from the VF module group corresponding to each PF module in the PF module group, The target VF module group.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and further perform the following operations: determining that the first PF module is re-enabled or failback, and the first PF module is The PF module in the PF module group is determined; based on the first PF module, the VF module group corresponding to the first PF module is established.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and further perform the following operations: determining that the second PF module is deleted from the PF module group of the first physical network card; releasing the second PF The VF module group corresponding to the module.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and further perform the following operations: when determining that the third PF module of the first physical network card fails, determining a VF corresponding to the third PF module Whether the VF module in the module group is available; when at least one VF module in the VF module group corresponding to the third PF module is available, determining the at least one VF module available in the VF module group corresponding to the first PF module To continue to be available.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and further perform the following operations: determining that the first VF module allocated to the first VM of the at least one VM in the target VF module group appears The second VF module different from the first VF module is selected from the VF module group corresponding to each PF module in the PF module group; and the target VF module group is updated.
  • the first physical NIC occupies at least two high-speed peripheral components interconnecting a standard PCIe bus
  • the processor 310 is configured to invoke the program code stored in the memory 320, and performs the following operations: determining that the target needs to be And transmitting, by the VF module group, the data of the VF module connected to the first PCIe bus of the at least two PCIe buses to the VF module of the target VF module group and the second PCIe bus of the at least two PCIe buses;
  • the data of the VF module connected to the first PCIe bus is sent to the VF module connected to the second PCIe bus through the switch of the first physical network card.
  • the processor 310 is configured to invoke the program code stored in the memory 320, and further perform the following operations: determining that the primary PF module is faulty; and selecting, from the PF module group, other than the primary PF module The PF module is used as a new main PF module; based on the VF module group corresponding to each PF module in the PF module group, a new target VF module group managed by the new main PF module is determined, wherein the new target VF module group The VF module is configured to be allocated to at least one virtual machine VM, so that the at least one VM implements data transmission.
  • a physical network card has a PF module group composed of at least two PF modules, and one PF module can be selected as a main PF module from at least two PF modules in the PF module group, thereby Due to the many opportunities to choose, you can avoid system failures. And reduce waste of resources.
  • the physical network card 400 includes: a PF module group 410 composed of at least two PF modules 411, and at least two VF module groups 420; wherein one of the at least two PF modules 411 is a PF module. 411 corresponds to one of the at least two VF module groups 420, and any one of the at least two PF modules 411 can manage other after being determined to be the main PF module.
  • the VF module 421 in the VF module group 420 to which the PF module 411 is connected is connected.
  • each PF module 411 in the PF module group 410 is connected to each other, and any PF module 411 in the PF module group 410 can manage the VF module group connected to the other PF module 411 through other PF modules 411.
  • VF module 421 in 420 is connected to each other, and any PF module 411 in the PF module group 410 can manage the VF module group connected to the other PF module 411 through other PF modules 411.
  • the VF module 421 in each of the at least two VF module groups 420 includes two types of spatial resources; wherein, one of the two types of spatial resources is used in the Before the primary PF module is determined, the PF module 411 corresponding to the VF module group 420 to which the VF module 421 to which the two types of spatial resources belong belongs is accessed; another type of spatial resource in the two types of spatial resources is used in the After the primary PF module is determined, establishing a connection with the primary PF module is accessed by the primary PF module.
  • the physical network card 400 occupies at least two high-speed peripheral components interconnecting a standard PCIe bus, and the physical network card 400 further includes a built-in switch; wherein the built-in switch is used to connect with the at least two PCIe buses The data of the VFE module connected to the first PCIe bus is transmitted to the VF module connected to the second PCIe bus of the at least two PCIe buses.
  • the physical network card is connected to the external switch by using at least two ports; the physical network card is further configured to receive a data stream sent by the device for data transmission through the aggregation link, and pass each port of the at least two ports. Equalizing the data stream to the external switch.
  • a physical network card has a PF module group composed of at least two PF modules, and one PF module can be selected as a main PF module from at least two PF modules in the PF module group, thereby Due to the many opportunities for selection, system failures can be avoided and resource waste can be reduced.
  • FIG. 7 is a schematic block diagram of a system 500 for data transmission in accordance with an embodiment of the present invention.
  • the system 400 includes a device 410 for data transmission and a physical network card 420; wherein the device 510 for data transmission may correspond to the device 200 or 300 for data transmission in the embodiment of the present invention, physical
  • the network card 520 can correspond to the physical network card 400, and the device 200 for data transmission
  • the first physical network card mentioned in 300 or 300 is the physical network card 400. For the sake of cleanliness, we will not repeat them here.
  • a physical network card has a PF module group composed of at least two PF modules, and one PF module can be selected as a main PF module from at least two PF modules in the PF module group, thereby Due to the many opportunities for selection, system failures can be avoided and resource waste can be reduced.
  • the disclosed systems, devices, and methods may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not executed.
  • the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be electrical, mechanical or otherwise.
  • the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a standalone product. Based on such understanding, the technical solution of the present invention, or the part contributing to the prior art, or the part of the technical solution, may be embodied in the form of a software product stored in a storage medium.
  • a number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like, which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Hardware Redundancy (AREA)
  • Computer And Data Communications (AREA)

Abstract

Le mode de réalisation de la présente invention concerne un procédé, un appareil et un système de transmission de données, et une carte de réseau physique. Le procédé comprend : déterminer un module de fonction physique (PF) principal dans un groupe de modules de fonction physique (PF) d'une première carte de réseau physique, le groupe de modules PF comprenant au moins deux modules PF; déterminer une grappe de modules de fonction virtuelle (VF) cible gérée par le module PF principal sur la base des groupes de modules de fonction virtuelle (VF) correspondant à chacun des modules PF dans le groupe de modules PF; et allouer des modules VF de la grappe de modules VF cible à au moins une machine virtuelle, ce qui permet à au moins une machine virtuelle de réaliser une transmission de données. Le procédé, l'appareil et le système de transmission de données dans le mode de réalisation de la présente invention permettent d'éviter les défaillances du système et de réduire le gaspillage des ressources.
PCT/CN2013/077412 2013-06-18 2013-06-18 Procédé, appareil et système de transmission de données, et carte de réseau physique WO2014201623A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201380000801.1A CN103609077B (zh) 2013-06-18 2013-06-18 用于数据传输的方法、装置和系统以及物理网卡
PCT/CN2013/077412 WO2014201623A1 (fr) 2013-06-18 2013-06-18 Procédé, appareil et système de transmission de données, et carte de réseau physique

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2013/077412 WO2014201623A1 (fr) 2013-06-18 2013-06-18 Procédé, appareil et système de transmission de données, et carte de réseau physique

Publications (1)

Publication Number Publication Date
WO2014201623A1 true WO2014201623A1 (fr) 2014-12-24

Family

ID=50126079

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2013/077412 WO2014201623A1 (fr) 2013-06-18 2013-06-18 Procédé, appareil et système de transmission de données, et carte de réseau physique

Country Status (2)

Country Link
CN (1) CN103609077B (fr)
WO (1) WO2014201623A1 (fr)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105591780B (zh) * 2014-10-24 2019-01-29 新华三技术有限公司 集群监测方法和设备
CN104486234B (zh) * 2014-11-21 2018-10-30 华为技术有限公司 一种将业务交换机卸载到物理网卡的方法及服务器
US9588791B2 (en) 2015-04-27 2017-03-07 Altera Corporation Flexible physical function and virtual function mapping
WO2017004746A1 (fr) * 2015-07-03 2017-01-12 华为技术有限公司 Procédé de configuration de réseau, système et dispositif de réseau
CN106484492A (zh) * 2015-08-28 2017-03-08 杭州华为数字技术有限公司 配置接口的方法和系统
CN106656539A (zh) * 2015-11-04 2017-05-10 中兴通讯股份有限公司 一种网口状态检测和同步方法及装置
CN107181679A (zh) * 2016-03-09 2017-09-19 中兴通讯股份有限公司 一种端口绑定实现方法及装置
SG11201800020UA (en) 2016-11-09 2018-06-28 Huawei Tech Co Ltd Packet processing method in cloud computing system, host, and system
JP6513835B2 (ja) 2016-11-09 2019-05-15 ホアウェイ・テクノロジーズ・カンパニー・リミテッド クラウドコンピューティングシステムにおけるパケット処理方法、ホスト、およびシステム
CN107643938A (zh) * 2017-08-24 2018-01-30 中国科学院计算机网络信息中心 数据传输方法、装置及存储介质
CN113132200B (zh) * 2019-12-30 2024-01-19 中兴通讯股份有限公司 数据转发方法、转发器、系统、服务器和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101436165A (zh) * 2007-11-15 2009-05-20 国际商业机器公司 用于iov适配器管理的系统和方法
US20110202702A1 (en) * 2010-02-17 2011-08-18 Frankie Fan Method and system for pci hybrid function
US20120167085A1 (en) * 2010-12-28 2012-06-28 Plx Technology, Inc. Sharing multiple virtual functions to a host using a pseudo physical function
CN102946366A (zh) * 2012-11-12 2013-02-27 杭州华为数字技术有限公司 带内管理方法及系统

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7984454B2 (en) * 2006-12-19 2011-07-19 International Business Machines Corporation Migration of single root stateless virtual functions
JP5232602B2 (ja) * 2008-10-30 2013-07-10 株式会社日立製作所 ストレージ装置、及びストレージコントローラ内部ネットワークのデータ経路フェイルオーバー方法
JP2012048546A (ja) * 2010-08-27 2012-03-08 Hitachi Ltd 計算機システム、ｉ／ｏデバイス制御方法、及びｉ／ｏドロワ

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101436165A (zh) * 2007-11-15 2009-05-20 国际商业机器公司 用于iov适配器管理的系统和方法
US20110202702A1 (en) * 2010-02-17 2011-08-18 Frankie Fan Method and system for pci hybrid function
US20120167085A1 (en) * 2010-12-28 2012-06-28 Plx Technology, Inc. Sharing multiple virtual functions to a host using a pseudo physical function
CN102946366A (zh) * 2012-11-12 2013-02-27 杭州华为数字技术有限公司 带内管理方法及系统

Also Published As

Publication number Publication date
CN103609077A (zh) 2014-02-26
CN103609077B (zh) 2017-02-22

Similar Documents

Publication Publication Date Title
WO2014201623A1 (fr) Procédé, appareil et système de transmission de données, et carte de réseau physique
US10986171B2 (en) Method for unified communication of server, baseboard management controller, and server
US9031081B2 (en) Method and system for switching in a virtualized platform
US11005755B2 (en) Packet processing method in cloud computing system, host, and system
US11086650B2 (en) Technologies for application-specific network acceleration with unified coherency domain
US8677023B2 (en) High availability and I/O aggregation for server environments
CA2991208C (fr) Methode de traitement par paquet dans un systeme informatique nuagique, hote et systeme
US20100287262A1 (en) Method and system for guaranteed end-to-end data flows in a local networking domain
JP4620776B2 (ja) ネットワークに付属する物理的なポートの仮想インスタンスを管理するための方法およびシステム
TWI515572B (zh) 分散式虛擬橋接器環境中暫存器存取之系統及方法
JP2015521779A (ja) 入力／出力仮想化のためのシステムおよび方法
US9727386B2 (en) Method and apparatus for network resource virtual partitioning
WO2018133035A1 (fr) Procédé, carte réseau, dispositif hôte et système informatique pour transférer des paquets de données
EP3629162B1 (fr) Technologies de séparation de plan de commande au niveau d'un contrôleur d'interface réseau
CA2961283C (fr) Procede de gestion de ressources, hote, et point d'extremite
US20150113114A1 (en) Network interface adapter registration method, driver, and server
WO2014206105A1 (fr) Procédé de commutateur virtuel, appareil s'y rapportant et système informatique
US11263162B2 (en) System decoder for training accelerators
WO2016107023A1 (fr) Système serveur en nuage
WO2013086861A1 (fr) Procédé d'accès à un équipement d'entrée/sortie (e/s) multichemin, et gestionnaire et système multichemin d'e/s
JP2012519340A (ja) 仮想ネットワーク環境におけるコピーの迂回

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13887489

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13887489

Country of ref document: EP

Kind code of ref document: A1