CN113904938B - System and method for dynamically configuring PCIe terminal equipment - Google Patents

System and method for dynamically configuring PCIe terminal equipment Download PDF

Info

Publication number
CN113904938B
CN113904938B CN202111140694.5A CN202111140694A CN113904938B CN 113904938 B CN113904938 B CN 113904938B CN 202111140694 A CN202111140694 A CN 202111140694A CN 113904938 B CN113904938 B CN 113904938B
Authority
CN
China
Prior art keywords
pcie
configuration
message
module
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111140694.5A
Other languages
Chinese (zh)
Other versions
CN113904938A (en
Inventor
王昕溥
王伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dayu Zhixin Technology Co ltd
Original Assignee
Beijing Dayu Zhixin Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dayu Zhixin Technology Co ltd filed Critical Beijing Dayu Zhixin Technology Co ltd
Priority to CN202111140694.5A priority Critical patent/CN113904938B/en
Publication of CN113904938A publication Critical patent/CN113904938A/en
Application granted granted Critical
Publication of CN113904938B publication Critical patent/CN113904938B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08Configuration management of networks or network elements
    • H04L41/0893Assignment of logical groups to network elements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02Standardisation; Integration
    • H04L41/0246Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08Configuration management of networks or network elements
    • H04L41/0803Configuration setting
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14Network analysis or design
    • H04L41/145Network analysis or design involving simulating, designing, planning or modelling of a network

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a system and a method for dynamically configuring PCIe terminal equipment, wherein the system comprises the following steps: the PCIe virtualization module, the external debugger and the embedded processor further simulate a plurality of terminal devices connected with downstream ports of the PCIe switch device by simulating the PCIe switch device, provide development support capable of external debugging and internal programming, and realize dynamic configuration and unified management of the PCIe terminal devices by simulating the PCIe switch; an external debugging interface, an internal embedded processor and a programmable state machine adopt a method of joint control by stages and tasks, thereby effectively meeting the requirements of comprehensively, flexibly and real-timely responding and processing PCIe transaction layer messages under the conditions of debugging, application, updating and abnormal conditions; corresponding functional design is carried out in an IO processing module in the PCIe virtualization module to support two IO operation modes of equipment IO and host IO, and the IO operation efficiency is improved.

Description

System and method for dynamically configuring PCIe terminal equipment
Technical Field
The invention relates to the field of cloud computing, in particular to a system and a method for dynamically configuring PCIe terminal equipment.
Background
In the current data center application scenario, the elastic bare metal server gradually takes more weight, and a user rents a physical server in an elastic manner similar to a cloud service. Different from a traditional server leasing mode, the elastic bare metal server provides complete IO resources and has flexibility and configurability of cloud services. The dynamic configuration of the IO resources of the user, such as the network equipment, the storage equipment, the computing acceleration equipment and the like, can be immediately implemented on the rented elastic bare metal server.
The multiplexing of PCIe terminal devices mostly adopts an SR-IOV (Single Root I/O Virtualization) mode, virtualizes a plurality of VFs (Virtual Functions) by using one PF (Physical Functions) as a template, provides Virtualization support from the perspective of the terminal device, and has high efficiency of hardware implementation. Due to the lack of requirements of virtualization management on the dynamic and flexible configuration of the attributes and the number of the terminal devices from the hardware perspective. The number and the attribute of the equipment are both statically set, and the equipment cannot be modified online, if the VF is modified, all the VFs under the same PF can be influenced, and the requirement that the dynamic configuration of the elastic bare metal server to the PCIe equipment takes effect instantly cannot be met. In addition, when providing support for dynamic configuration of PCIe devices in a data center application scenario based on an elastic bare metal server, how to improve the efficiency of IO processing is also an urgent problem to be solved.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method and a system for labeling a medical text entity, which specifically include:
in a first aspect, an embodiment of the present invention provides a system for dynamically configuring PCIe terminal devices, where the system includes:
the PCIe virtualization module is used for receiving a PCIe configuration request message sent by a host when the host scans PCIe terminal equipment at the downstream of the system and replying a configuration response message to the configuration request message; the PCIe IO request message is used for receiving a PCIe IO request message sent by the host, and an IO response message is replied to the configuration request message; the PCIe IO request message comprises a BAR address space memory read-write request corresponding to downstream PCIe terminal equipment;
the external debugger is connected with the PCIe virtualization module through a debugging interface provided by the PCIe virtualization module and used for replacing the PCIe virtualization module to debug PCIe configuration and IO call through the debugging interface provided by the PCIe virtualization module during starting; the system comprises a PCIe virtualization module, a target type PCIe terminal device and a program, wherein the PCIe virtualization module is used for modifying the program used for supporting the target type PCIe terminal device in the PCIe virtualization module after the target type PCIe terminal device is debugged;
and the embedded processor is used for negotiating with an external configuration management server to generate the configuration response message and determining the generation sequence of the plurality of configuration response messages.
Optionally, the PCIe virtualization module includes:
the PCIe transaction layer message distributor is used for analyzing a PCIe transaction layer message received from a PCIe link layer and respectively sending the PCIe configuration transaction request message or the PCIe IO transaction request message to a corresponding configuration processing module or an IO processing module according to a PCIe configuration transaction request message or a PCIe IO transaction request message indicated by message header information of the PCIe transaction layer message; the PCIe transaction layer message is used for distributing the PCIe transaction layer message received from the configuration processing module or the IO processing module to a PCIe link layer;
the debugging interface is connected with the external debugger and used for sending and receiving PCIe transaction layer messages by replacing the PCIe transaction layer message distributor when the external debugger is started;
the configuration processing module is used for receiving the configuration message and finishing updating the PCIe terminal equipment according to the configuration message;
and the IO processing module is used for receiving a PCIe IO request message and controlling the PCIe virtualization module and/or at least one PCIe terminal device to access a control state register and a direct memory according to the PCIe IO request message.
Optionally, the configuration processing module includes:
the configuration message receiving and sending interface is used for receiving PCIe configuration transaction request messages distributed by the PCIe transaction layer message distributor and sending configuration transaction response messages received from the programmable hardware state machine or the CPU configuration processing interface;
the CPU configuration processing interface is used for receiving PCIe configuration transaction request messages forwarded by the configuration message transceiving interface and sending configuration transaction response messages received from the embedded processor;
the programmable hardware state machine is used for receiving and analyzing the PCIe configuration transaction request message forwarded by the configuration message transceiving interface and sending a pre-programmed configuration transaction response message according to the analysis content;
and the state machine micro-sequence program storage unit is used for storing instructions and data of the programmable hardware state machine, and the instructions and the data are written in or read out from the CPU configuration processing interface.
Optionally, the IO processing module includes:
the target IO message receiving and sending interface is used for receiving PCIe transaction request messages forwarded by the PCIe transaction layer message distributor and analyzing the PCIe transaction request messages into BAR address space memory read-write requests; the memory read-write response returned by the CSR module is encapsulated into an IO transaction response message and sent to a PCIe transaction layer message distributor;
and the CSR module is used for receiving the BAR address space memory read-write request sent by the target IO message transceiving interface, acquiring corresponding read-write request content according to the BAR address space memory read-write request and sending the read-write request content to the target IO message transceiving interface.
Optionally, the CSR module comprises:
the BAR address space mapping management interface is used for providing a BAR address corresponding to a target function of corresponding PCIe terminal equipment after receiving a BAR address space memory read-write request;
and the BAR address space memory is used for storing and providing the content of the BAR address space.
Optionally, the CSR module further includes a BAR specific address access trigger, configured to trigger a programmable state machine of the IO processing module to run a BAR address space memory update program and a DMA request program when the target BAR address is accessed;
correspondingly, the IO processing module further includes:
the programmable state machine is used for responding to a BAR specific address access trigger signal sent by the BAR specific address access trigger and sending the update information of the address space memory of the BAR to the CSR module; the DMA module is also used for sending a DMA request to the DMA module and receiving a completion response of the DMA module;
the starting IO message transceiving interface is used for receiving a host memory address space memory read-write request sent by the DMA module, packaging the host memory address space memory read-write request into an IO transaction request message and sending the IO transaction request message to the PCIe transaction layer message distributor; and is used for analyzing the IO transaction response message returned by the PCIe transaction layer message distributor into a host memory address space memory read-write response and sending the host memory address space memory read-write response to the DMA module;
the DMA module is used for responding to a DMA request of the programmable state machine, initiating a command to acquire DMA or data transmission DMA operation, and sending a read-write request of a memory address space memory of the host to the initial IO message transceiving interface; and writing the read-write response content of the memory in the memory address space of the host computer received from the initial IO message transceiving interface into a command cache or a data cache.
In a second aspect, an embodiment of the present invention provides a method for dynamically configuring PCIe terminal equipment, where the method is applied to the system in the first aspect, and is characterized in that the method includes:
a PCIe virtualization module of the system receives a PCIe configuration request message sent by a host when the host scans PCIe terminal equipment at the downstream of the system;
a configuration processing module of the PCIe virtualization module replies a PCIe configuration response message to the PCIe configuration request message, wherein the configuration response message is generated by negotiation between an embedded processor of the system and an external configuration management server, and the generation sequence of the configuration response messages is determined by the embedded processor;
a PCIe virtualization module of the system receives a PCIe IO request message sent by the host, wherein the PCIe IO request message comprises a BAR address space memory read-write request corresponding to downstream PCIe terminal equipment;
and an IO processing module of the PCIe virtualization module replies an IO response message to the PCIe IO request message, wherein the IO response message comprises the content stored in the BAR address space memory.
Optionally, the method further comprises:
when the target BAR address is accessed, the programmable state machine of the IO processing module is triggered to run a BAR address space memory update program and a DMA request program.
Optionally, the method further comprises:
starting an external debugger of the system;
the external debugger replaces the PCIe virtualization module to debug PCIe configuration and IO call through a debugging interface provided by the PCIe virtualization module;
and after the target type PCIe terminal equipment is debugged, modifying a program which is used for supporting the target type PCIe terminal equipment in the PCIe virtualization module.
Optionally, the method further comprises:
replacing a negotiation process of the embedded processor and an external configuration management server by a programmable state machine of a configuration processing module of the PCIe virtualization module to generate the configuration response message;
and if the PCIe configuration request message is of an undefined type, generating the configuration response message instead of the programmable state machine according to the negotiation between the embedded processor and an external configuration management server.
The system and the method for dynamically configuring the PCIe terminal equipment provided by the embodiment of the invention further simulate a plurality of terminal equipment connected with the downstream port of the PCIe switch equipment by simulating the PCIe switch equipment, provide development support which can be debugged externally and can be programmed internally, and realize the dynamic configuration and the unified management of the PCIe terminal equipment by simulating the PCIe switch; an external debugging interface, an internal embedded processor and a programmable state machine adopt a method of joint control by stages and tasks, thereby effectively meeting the requirements of comprehensively, flexibly and real-timely responding and processing PCIe transaction layer messages under the conditions of debugging, application, updating and abnormal conditions; corresponding functional design is carried out in an IO processing module in the PCIe virtualization module to support two IO operation modes of equipment IO and host IO, and the IO operation efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative work. The foregoing and other objects, features and advantages of the application will be apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the drawings. The drawings are not intended to be to scale as practical, emphasis instead being placed upon illustrating the subject matter of the present application.
Fig. 1 is a schematic structural diagram illustrating a system for dynamically configuring a PCIe terminal device according to an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a PCIe virtualization module provided in an embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a configuration processing module according to an embodiment of the present invention.
Fig. 4 shows a schematic structural diagram of an IO processing module according to an embodiment of the present invention.
Fig. 5 is a schematic diagram illustrating a complete structure of a system for dynamically configuring a PCIe terminal device according to an embodiment of the present invention.
Fig. 6 is a flowchart illustrating a method for dynamically configuring a PCIe terminal device according to an embodiment of the present invention.
Fig. 7 is a flowchart illustrating a complete flow chart of a method for dynamically configuring a PCIe terminal device according to an embodiment of the present invention.
Fig. 8 is a flowchart illustrating a debugging method of a PCIe terminal device according to an embodiment of the present invention.
Fig. 9 is a flowchart illustrating an exception handling method for dynamically configuring a PCIe terminal device according to an embodiment of the present invention.
Detailed Description
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It is to be understood that such description is merely illustrative and not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The words "a", "an" and "the" and the like as used herein are also intended to include the meanings of "a plurality" and "the" unless the context clearly dictates otherwise. Furthermore, the terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
In the current data center application scenario, the elastic bare metal server gradually takes more weight, and a user rents a physical server in an elastic manner similar to a cloud service. Different from a traditional server leasing mode, the elastic bare metal server provides complete IO resources and has flexibility and configurability of cloud services. The dynamic configuration of the IO resources of the user, such as the network equipment, the storage equipment, the computing acceleration equipment and the like, can be immediately implemented on the rented elastic bare metal server.
The multiplexing of PCIe terminal devices mostly adopts an SR-IOV (Single Root I/O Virtualization) mode, virtualizes a plurality of VFs (Virtual Functions) by using one PF (Physical Functions) as a template, provides Virtualization support from the perspective of the terminal device, and has high efficiency of hardware implementation. Due to the lack of requirements of virtualization management on the dynamic and flexible configuration of the attributes and the number of the terminal devices from the hardware perspective. The number and the attribute of the equipment are both statically set, and the equipment cannot be modified online, if the VF is modified, all the VFs under the same PF can be influenced, and the requirement that the dynamic configuration of the elastic bare metal server to the PCIe equipment takes effect instantly cannot be met. In addition, when providing support for dynamic configuration of PCIe devices in a data center application scenario based on an elastic bare metal server, how to improve the efficiency of IO processing is also an urgent problem to be solved.
In view of the above, embodiments of the present invention are directed to a system and a method for dynamically configuring a PCIe endpoint device, and the disclosure of the embodiments of the present invention is described in detail below with reference to the drawings.
Fig. 1 is a schematic structural diagram illustrating a system for dynamically configuring a PCIe terminal device according to an embodiment of the present invention, which is described in detail below.
A PCIe virtualization module 110, configured to receive a PCIe configuration request packet sent by a host when the host scans PCIe terminal devices downstream of the system, and reply a configuration response packet to the configuration request packet; the PCIe IO request message is used for receiving a PCIe IO request message sent by the host, and an IO response message is replied to the configuration request message; the PCIe IO request message comprises a BAR address space memory read-write request corresponding to downstream PCIe terminal equipment.
Because the technology of providing virtualization support for multiplexing of PCIe terminal equipment from the hardware perspective based on the application scenario of the elastic bare metal server data center in the prior art is used, the system for dynamically configuring PCIe terminal equipment in the embodiment of the invention is a software function system arranged on the equipment, aiming at the problem that the dynamic flexible configuration of the attribute and the quantity of the terminal equipment by virtualization management cannot support.
The system for dynamically configuring the PCIe terminal equipment consists of three main modules, namely a PCIe virtualization module 110, an external debugger 120 and an embedded processor 130. The PCIe virtualization module 110 may not only simulate the function of the PCIe switch device from software, but also further simulate a plurality of terminal devices connected to the downstream ports thereof.
When the host boots up, the topology of downstream PCIe devices is scanned during the PCIe enumeration phase. When the equipment where the system is located is scanned, the PCIe virtualization module responds to be a PCIe switch; the host further scans the downstream PCIe devices of the device, and the PCIe virtualization module can replace the virtual downstream PCIe device to respond, which is represented by a logical PCIe switch and a plurality of downstream PCIe terminal devices in various logics. After the host finishes power-on startup and PCIe device scanning, the host enters a running stage, the device where the system is located can initiate a PCIe hot plug request, and multiple or multiple logical PCIe terminal devices can be deleted or added dynamically. The number and the attribute of logic PCIe terminal equipment at the downstream of the equipment of the system can be flexibly configured, and the configuration content is negotiated and determined by the configuration management server and the embedded processor of the equipment when the equipment is started or operated. In an application scenario of the data center based on the elastic bare metal server, the downstream logical PCIe terminal device may be any device type deployed on a PCI slot, such as a sound card, a network card, a built-in Modem, and the like.
Specifically, in the host configuration stage, the host scans the downstream PCIe terminal device in the system and performs corresponding configuration work, in the embodiment of the present invention, the PCIe virtualization module 110 replaces the PCIe terminal device, receives a PCIe configuration request message sent by the host, and replies a configuration response message to the configuration request message. That is, the host is implemented by the PCIe virtualization module 110 when scanning and configuring the PCIe endpoint devices. The PCIe configuration request message and the configuration response message are message types used for equipment information interaction when the PCIe configuration request message and the configuration response message are used for configuring PCIe terminal equipment in a PCIe protocol.
In addition, in the device IO operation stage, the host may request to access the configured IO resource corresponding to the PCIe terminal device. At this time, the embodiment of the present invention is also responsible for receiving, through the PCIe virtualization module 110, a PCIe IO request packet sent by the host, and replying an IO response packet to the configuration request packet. The PCIe IO request message and the IO response message are message types used for device information interaction when accessing PCIe terminal device resources in a PCIe protocol. The PCIe IO request message comprises a BAR address space memory read-write request corresponding to downstream PCIe terminal equipment. The BAR address space is composed of a plurality of BAR addresses corresponding to a plurality of functions representing a plurality of PCIE terminal devices, and the data stored in the BAR address space memory represents base addresses of the PCIE terminal devices in the PCIE address space. Therefore, through the PCIe IO request packet including the BAR address space memory read-write request, the host may request to access the IO resource and the corresponding function corresponding to the address.
According to the description content of the functions of the PCIe virtualization module 110, it can be seen that, in the PCIe virtualization module 110 in the embodiment of the present invention, no matter in the host configuration stage or the device IO operation stage, the functions of simulating PCIe switch devices and further simulating multiple terminal devices connected to downstream ports of the PCIe switch devices are implemented by performing configuration/IO message interaction with the host.
The external debugger 120 is connected with the PCIe virtualization module through a debug interface provided by the PCIe virtualization module, and is configured to perform, instead of the PCIe virtualization module, PCIe configuration and IO call debugging through the debug interface provided by the PCIe virtualization module when starting up; and after the target type PCIe terminal equipment is debugged, modifying a program in the PCIe virtualization module for supporting the target type PCIe terminal equipment.
The system for dynamically configuring the PCIe terminal device provided in the embodiment of the present invention further has the programmable and dynamic online updating capability through the settings of the external debugger 120 and the PCIe virtualization module 110, and can support virtual implementation of the existing PCIe device and a PCIe device of a new type that may appear in the future. The support for different types of PCIe devices is realized by adding or modifying the program storage content of the device where the system is located, the system can simultaneously support a plurality of different types of PCIe devices, and the types of the support devices are limited by the program storage capacity. Aiming at the new type of PCIe equipment which is not realized in the system, the system provides an external debugger 120, a corresponding debugging interface is arranged in the PCIe virtualization module 110 to be connected with the external debugger 120, the external debugger 120 can take over the function of processing the configuration/IO message of the PCIe virtualization module 110 when the debugging function is operated, and then the external debugger host replaces the PCIe virtualization module 110 to complete the function simulation of a PCIe switch and downstream PCIe terminal equipment. After the PCIe terminal device of the new type completes all functions in the external host debug mode, the corresponding program storage content in the PCIe virtualization module 110 may be modified, and the support for the PCIe terminal device of the new type is updated.
In the present system, since the external debugger 120 has the highest priority, it can be determined whether the external debugger is connected after the device in which the present system is located is powered on. If so, the system enters a debug state. When the external debugger is started to carry out debugging, the external debugger can replace the PCIe virtualization module to carry out debugging of PCIe configuration and IO calling through a debugging interface provided by the PCIe virtualization module, and modification of existing PCIe equipment and updating of new PCIe equipment are achieved. After the debugging is completed, the corresponding program storage content in the PCIe virtualization module 110 is modified in a programmable manner, so that support for a new type of PCIe terminal device is achieved.
Through the process, the system realizes the dynamic update of the number or the attributes of the PCIe terminal devices. When the PCIe terminal equipment needs to be updated, an external debugger is started to intervene, the PCIe terminal equipment can be dynamically configured, and meanwhile, the problem that the configuration flexibility is poor when the traditional SR-IOV method is used for multiplexing the PCIe terminal equipment is solved.
The embedded processor 130 is configured to negotiate with an external configuration management server to generate the configuration response packet, and determine a generation sequence of the configuration response packets.
The embedded processor 130 provided in the embodiment of the present invention is responsible for interacting with an external configuration management server (not shown in the figure), responding to a configuration update request and updating configuration contents, so as to flexibly simulate multiple PCIe terminal devices and multiple PCIe terminal devices, and dynamically add and delete multiple PCIe terminal devices during the host operation. The embedded processor may be connected to the configuration management server through an ethernet interface. The embedded processor can realize the simulation of the PCIe terminal equipment independently, and can also realize the simulation of the PCIe terminal equipment by matching with the configuration processing module and the IO processing module.
Specifically, the PCIe virtualization module 110 in the normal flow and the external debugger 120 in the debug flow described above have functions of distributing PCIe configuration request messages and configuration response messages, respectively. But wherein the generation of the configuration reply message is negotiated by the embedded processor 130 with the external configuration management server. In addition, the embedded processor 130 further stores an internal program, which is used to determine a generation sequence of the configuration response messages, and is applicable to a PCIe configuration request queue formed for a plurality of PCIe configuration requests, and the generation sequence of the configuration response messages corresponding to the PCIe configuration request queue is determined according to a processing logic preset in the internal program stored in the embedded processor 130. The processing logic preset in the internal program stored in the embedded processor 130 may determine the generation sequence of the configuration response message according to the arrival time sequence of the PCIe configuration request message, or may determine the generation sequence of the configuration response message according to the importance of the requested PCIe terminal device, which is not specifically limited in the embodiment of the present invention. The external configuration management server may be an external server device storing PCIe configuration information, and may acquire the latest PCIe configuration information by communicating with the external server device, and the external configuration management server may be provided by a third party hardware vendor.
It can be seen that the embedded processor 130 provides a function similar to a CPU in the present system, and realizes the generation of the message content itself and the determination of the message generation sequence in the configuration response process through the communication with the external configuration management server. After the external configuration management server updates the configuration information, the embedded processor 130 obtains the latest configuration information through negotiation with the external configuration management server, and realizes generation of the message content itself and determination of the message generation sequence based on the latest configuration information.
The relationship among the PCIe virtualization module 110, the external debugger 120, and the embedded processor 130 is: the PCIe virtualization module 110 is immediately started after being powered on, and controls the configuration and IO processing work of the PCIe terminal device as a role of a simulated switch in a conventional flow. The function corresponding to the external debugger 120 has the highest priority, and may monitor and close the other two interfaces, and only enable the function in the debug state, instead of the PCIe virtualization module 110, perform the configuration of the PCIe terminal device and the debugging work of the IO processing. After the embedded processor 130 works, the configuration response message may be generated by means of, for example, switching a jump address of a programmable hardware state machine according to the latest configuration of the external configuration management server, and the generation sequence of the configuration response messages may be determined. The external debugger may be a processor on the same circuit board as the device in which the system is located, or may be a processor outside the circuit board (for example, a PC for debugging). The embedded processor is a special processor on the same circuit board of the device where the system is located.
The system for dynamically configuring the PCIe terminal equipment provided by the embodiment of the invention simulates PCIe switch equipment through a PCIe virtualization module, further simulates a plurality of terminal equipment connected with a downstream port of the PCIe switch equipment, is responsible for generating and determining an actual configuration message through an embedded processor, and realizes development support of external debugging and internal programming through the setting of an external debugger, thereby effectively meeting the requirements of flexible configuration and application of the PCIe terminal equipment under the conditions of application, debugging, updating and the like through the overall design of the system.
Based on the foregoing embodiments, fig. 2 shows a schematic structural diagram of a PCIe virtualization module provided in the embodiment of the present invention, and the details are as follows.
A PCIe transaction layer message distributor 101, configured to analyze a PCIe transaction layer message received from a PCIe link layer, and send the PCIe configuration transaction request message or the PCIe IO transaction request message to a corresponding configuration processing module or IO processing module according to a PCIe configuration transaction request message or a PCIe IO transaction request message indicated by message header information of the PCIe transaction layer message; and the PCIe transaction layer message is used for distributing the PCIe transaction layer message received from the configuration processing module or the IO processing module to a PCIe link layer.
The PCIe virtualization module provided in the embodiment of the present invention is composed of main modules, such as a PCIe transaction layer message distributor 101, a debug interface 102, a configuration processing module 103, and an IO processing module 104, in fig. 2. In addition, the PCIe virtualization module further includes a conventional PCIe physical layer (not shown in the figure) responsible for processing physical signals and exchanging data with the PCIe link layer; the PCIe link layer (not shown) is responsible for the creation, decoding, and checking of data link layer packets. The PCIe physical layer and the PCIe link layer both conform to the PCIe protocol specification, which is not described in this embodiment of the present invention.
The PCIe transaction layer message processed by the PCIe link layer is sent to the PCIe transaction layer message distributor 101 for reception. And after receiving the PCIe transaction layer message of the PCIe link layer, the PCIe transaction layer message distributor only analyzes the message header and distinguishes the message as a PCIe configuration transaction request message or a PCIe IO transaction request message according to the message header information. If the transaction request message is the PCIe configuration transaction request message, the message is forwarded to the configuration processing module 103 for processing; if the transaction request message is the PCIe IO transaction request message, the transaction request message is forwarded to the IO processing module 104 for processing. Correspondingly, the response messages sent by the configuration processing module 103 and the IO processing module 104 are also sent to the PCIe link layer through the PCIe transaction layer message distributor. Because the PCIe virtualization module in the embodiment of the invention plays the role of a PCIe switch, a PCIe transaction layer message distributor is required to be arranged in the PCIe virtualization module, and the received message is distinguished to belong to a PCIe configuration transaction request message or a PCIe IO transaction request message according to the header formats of different types of messages in a PCIe protocol, so that the unified management of downstream PCIe terminal equipment is realized.
And the debugging interface 102 is connected with the external debugger and is used for sending and receiving PCIe transaction layer messages by replacing the PCIe transaction layer message distributor when the external debugger is started.
In the foregoing embodiment, the external debugger takes over the function of the PCIe virtualization module to process the configuration/IO packet, so that the external debugger host replaces the PCIe virtualization module 110 to complete the function simulation of the PCIe switch and the downstream PCIe terminal device. Specifically, in a debug state, the external debugger actually communicates with the debug interface 102 in the PCIe virtualization module, and the external debugger actively takes over the sending and receiving of the message from the PCIe transaction layer message distributor through the debug interface 102, or passively captures and forwards all transaction layer messages interacted between the PCIe transaction layer message distributor and the configuration processing module and the IO processing module.
The active takeover or passive capture mode is determined by the debugging process defined by the system, and the debugging interface can directly transmit messages with a PCIe link layer and can also automatically capture messages passing through a message distributor of a PCIe transaction layer. The debugging interface is controlled by an external debugger to process the message sending and receiving work of the message distributor of the PCIe transaction layer.
And the configuration processing module 103 is configured to receive the configuration message, and complete updating of the PCIe terminal device according to the configuration message.
The configuration processing module in the embodiment of the present invention is used for a PCIe device dynamic configuration function for which a PCIe switch is responsible and which is simulated by a PCIe virtualization module, and specifically includes: receiving and sending a PCIe configuration transaction request message defined in a PCIe protocol, responding to a power-on PCIe enumeration process of a host, initiating hot-plug operation and finishing the updating of the number and the attributes of PCIe terminal devices. By responding to the PCIe configuration transaction request packet, the PCIe virtualization module provided in the embodiment of the present invention may be represented as one PCIe terminal device, one PCIe switch device, or one PCIe switch device and a plurality of PCIe terminal devices connected downstream thereof through software function simulation.
The more detailed structure and function of the configuration processing module 103 will be further described in the following embodiments.
The IO processing module 104 is configured to receive a PCIe IO request packet, and control the PCIe virtualization module and/or the at least one PCIe terminal device to perform control status register access and direct memory access according to the PCIe IO request packet.
The IO processing module 104 provided in the embodiment of the present invention is configured to complete a PCIe device dynamic configuration function according to the configuration processing module, and then is responsible for completing a function of an IO terminal device access in an IO processing stage, and specifically includes: receiving and sending a PCIe IO request message defined in a PCIe protocol, responding to the Access of a control state register of a host driver, and initiating Direct Memory Access (DMA) of a device end. The system can realize independent control state registers of one PCIe virtualization module and/or a plurality of PCIe terminal devices according to the host configuration negotiation result; in addition, for a specific access object, the system can independently initiate direct memory access operation from one PCIe virtualization module and/or a plurality of PCIe terminal devices.
The more detailed structure and function of the IO processing module 104 will be further described in the following embodiments.
Based on any of the above embodiments, fig. 3 shows a schematic structural diagram of a configuration processing module provided by an embodiment of the present invention, which is specifically described as follows.
And the configuration message receiving and sending interface is used for receiving a PCIe configuration transaction request message distributed by the PCIe transaction layer message distributor and sending a configuration transaction response message received from the programmable hardware state machine or the CPU configuration processing interface.
When the message type distributed by the message distributor of the PCIe transaction layer in the embodiment of the invention is a PCIe configuration transaction request message, the message is sent to the configuration processing module, and the configuration processing module receives the message through the configuration message receiving and sending interface. Similarly, when the scheme in the embodiment of the present invention is in the debug state, the configuration processing module receives a PCIe configuration transaction request message distributed by the debug interface through the configuration message transceiving interface.
The configuration message transceiving interface needs to send a corresponding configuration transaction response message in addition to receiving the PCIe configuration transaction request message. The configuration response message in the embodiment of the invention can be generated by the programmable hardware state machine and then sent to the configuration message receiving and sending interface, or can be generated by the embedded processor and then sent to the configuration message receiving and sending interface through the CPU configuration processing interface.
And the CPU configuration processing interface is used for receiving the PCIe configuration transaction request message forwarded by the configuration message transceiving interface and sending a configuration transaction response message received from the embedded processor.
The main function of the CPU configuration processing interface in the embodiment of the present invention is an interface for communicating the PCIe virtualization module with the embedded processor. According to the embedded processor, the system is mainly responsible for generating the configuration response message, so the CPU configuration processing interface is mainly used for forwarding the PCIe configuration transaction request message to the embedded processor for processing, and receiving the response message from the embedded processor after the embedded processor generates the configuration transaction response message after finishing processing.
The programmable hardware state machine is used for receiving and analyzing the PCIe configuration transaction request message forwarded by the configuration message transceiving interface and sending a pre-programmed configuration transaction response message according to the analysis content;
the programmable hardware state machine of the configuration processing module in the embodiment of the invention also has the function of processing PCIe configuration transaction request messages. The programmable hardware state machine can start working after being electrified, can analyze the PCIe configuration transaction request message according to the pre-programmed configuration transaction response message, and can respond the most basic PCIe protocol, control the response of the PCIe protocol, suspend PCIe protocol interaction and the like. Meanwhile, the programmable hardware state machine can work together with the CPU configuration processing interface after the CPU configuration processing interface starts to work after being electrified.
Specifically, in the embodiment of the present invention, the configuration message may be separately received and sent by the CPU configuration processing interface or the programmable hardware state machine, or may be completed by the cooperation of the two. One possible way of cooperation is: the programmable hardware state machine is started immediately after being electrified, and after receiving the configuration read-write request of the upstream port, the programmable hardware state machine sends a pre-programmed message until the CPU completes the electrification initialization, and the CPU takes over the programmable state machine to complete the subsequent configuration read-write request response. Another possible way of cooperation is: the programmable state machine completes all preset read-write request responses, when an abnormal condition except the preset condition is met, the embedded processor is triggered to take over processing, and after the processing is completed, the programmable state machine still completes the request responses.
And the state machine micro-sequence program storage unit is used for storing instructions and data of the programmable hardware state machine, and the instructions and the data are written in or read out from the CPU configuration processing interface. The module can be understood as a data storage unit of a programmable hardware state machine, and when the CPU configuration processing interface writes data in the module, the CPU configuration processing interface can switch a jump address in a pre-programmed response message in the programmable hardware state machine.
The configuration processing module provided by the embodiment of the invention is matched with an external embedded processor through the set programmable state machine, and a staged and tasked combined control method is adopted, so that the support for the dynamic configuration of the PCIe equipment to take effect immediately is provided, the flexible processing requirement of PCIe configuration transactions is effectively met, and the unified management of the local PCIe terminal equipment and the downstream PCIe terminal equipment by simulating the PCIe switch equipment through the PCIe virtualization module where the configuration processing module is located is realized.
Based on any of the above embodiments, fig. 4 shows a schematic structural diagram of an IO processing module provided in an embodiment of the present invention, which is specifically described as follows.
The target IO message receiving and sending interface is used for receiving PCIe transaction request messages forwarded by the PCIe transaction layer message distributor and analyzing the PCIe transaction request messages into BAR address space memory read-write requests; and the memory read-write response returned by the CSR module is encapsulated into an IO transaction response message, and the IO transaction response message is sent to the PCIe transaction layer message distributor.
And the CSR module is used for receiving the BAR address space memory read-write request sent by the target IO message transceiving interface, acquiring corresponding read-write request content according to the BAR address space memory read-write request and sending the read-write request content to the target IO message transceiving interface.
The target IO message transceiving interface and the CSR module in the IO processing module provided in the embodiment of the present invention are used for operating a PCIe IO transaction request under a conventional condition, that is, a host IO phase transaction. The target IO message receiving and transmitting interface is mainly used for receiving and transmitting IO messages, and the CSR module is used for processing IO read-write transactions.
When the PCIe transaction layer message distributor in the embodiment of the present invention distributes a PCIe IO transaction request message, the message is sent to an IO processing module, specifically, a target IO message transceiving interface in the IO processing module, and the target IO message transceiving interface further parses the message into a BAR address space memory read-write request, and then distributes the read-write request to a CSR module to process a specific IO read-write transaction.
And after the CSR module returns the read-write response of the memory, the target IO message transceiving interface encapsulates the response content into an IO transaction response message and then sends the IO transaction response message to the PCIe transaction layer message distributor to complete the response process of the IO request.
The specific structure of the CSR module comprises:
and the BAR address space mapping management interface is used for providing a BAR address corresponding to a target function of the corresponding PCIe terminal equipment after receiving the BAR address space memory read-write request.
And the BAR address space memory is used for storing and providing the content of the BAR address space.
And the BAR specific address access trigger is used for triggering the programmable state machine of the IO processing module to run a BAR address space memory updating program and a DMA request program when the target BAR address is accessed.
The CSR module in the embodiment of the invention is used for specific operation of IO transactions under the conventional condition. Firstly, a BAR address space mapping management interface provides a BAR address corresponding to a target function of a corresponding PCIe terminal device according to a BAR address space memory read-write request analyzed by a target IO message transceiving interface, and a skilled person in the art can understand that the PCIe terminal device can specifically call a specific function of the PCIe terminal device in the form of a BAR address and in an address mapping mode when managing, once the value of the BAR address is determined, an internal storage space in the PCIe terminal device within an appointed range can be accessed, and when the PCIe terminal device confirms that an address in an IO request is within the range of the own BAR, the request can be accepted to realize IO operation. The mapping relation between the address represented by the BAR address space memory and the specific function of the PCIe terminal device may be pre-stored in the BAR address space memory, and provided to the BAR address space mapping management interface for use during IO operation.
The BAR specific address access trigger is used for triggering a programmable state machine of the IO processing module to run a BAR address space memory updating program and a DMA request program when a target BAR address is accessed, namely switching from a host IO stage under a conventional condition to an equipment IO stage related to a special BAR address is realized, and further realizing efficient IO operation corresponding to the specific BAR address in a DMA operation mode. The following will explain the relevant modules involved in the specific implementation process of the IO phase of the device and the specific functions thereof.
The programmable state machine is used for responding to a BAR specific address access trigger signal sent by the BAR specific address access trigger and sending the update information of the address space memory of the BAR to the CSR module; the DMA module is also used for sending a DMA request to the DMA module and receiving a completion response of the DMA module;
the starting IO message transceiving interface is used for receiving a host memory address space memory read-write request sent by the DMA module, packaging the host memory address space memory read-write request into an IO transaction request message and sending the IO transaction request message to the PCIe transaction layer message distributor; and is used for analyzing the IO transaction response message returned by the PCIe transaction layer message distributor into a host memory address space memory read-write response and sending the host memory address space memory read-write response to the DMA module;
the DMA module is used for responding to a DMA request of the programmable state machine, initiating a command to acquire DMA or data transmission DMA operation, and sending a read-write request of a memory address space memory of the host to the initial IO message transceiving interface; and writing the read-write response content of the memory in the memory address space of the host computer received from the initial IO message transceiving interface into a command cache or a data cache.
The programmable state machine, the initial IO message transceiving interface, and the DMA module in the IO processing module provided in the embodiment of the present invention are used to process an IO phase transaction of a device that needs to perform DMA operation when a BAR specific address access trigger signal is triggered. In the system provided by the embodiment of the present invention, in addition to the flexible configuration of the PCIe terminal device, the PCIe virtualization module needs to consider the efficiency of IO processing when simulating the PCIe switch and implementing the unified management of the PCIe terminal device. Therefore, when the IO processing module is designed to perform IO processing, in addition to the two modules of the target IO message transceiving interface and the CSR module being used for the operation of the PCIe IO transaction request under the conventional condition, the programmable state machine, the initial IO message transceiving interface, and the DMA module are also designed to be used for the IO processing of the device triggered under the special condition, and the two IO processing mechanisms are mutually matched to realize the efficient processing of IO.
The programmable state machine in the embodiment of the present invention is used for triggering device IO processing, and specifically, is used for responding to a BAR specific address access trigger signal sent by a BAR specific address access trigger in a CSR module, and sending address space memory update information of a BAR to the CSR module. And then, the programmable state machine sends a DMA request to the DMA module, completes the triggering process of the DMA operation, and receives the completion response of the DMA module after the DMA module completes the DMA operation.
Specifically, the programmable state machine works in an inquiry waiting state according to a preprogrammed micro-sequence program stored in a micro-sequence program storage unit of the state machine, and sequentially inquires a BAR specific address access trigger signal and a DMA module completion response signal according to a certain sequence and priority. Triggering and executing a corresponding micro-sequence program after receiving a BAR specific address access trigger signal; and triggering and executing a corresponding micro-sequence program after receiving a completion response signal of the DMA module.
The function of the initial IO message transceiving interface is similar to that of the target IO message transceiving interface, but the initial IO message transceiving interface is mainly used for message transceiving of IO transactions of equipment. Specifically, after receiving a host memory address space memory read-write request sent by a DMA module, encapsulating the host memory address space memory read-write request into an IO transaction request message, and sending the IO transaction request message to a PCIe transaction layer message distributor; in addition, after an IO transaction response message returned by the PCIe transaction layer message distributor is sent to the initial IO message transceiving interface, the initial IO message transceiving interface analyzes the response message into a host memory address space memory read-write response and sends the host memory address space memory read-write response to the DMA module, so that DMA read-write operation is realized.
The DMA module is used for responding to the request of the programmable state machine, initiating a command to acquire DMA or data transmission DMA operation, and sending a read-write request of a host memory address space memory to the initial IO message transceiving interface; and writing the read-write response content of the memory in the memory address space of the host computer received from the initial IO message transceiving interface into a command cache or a data cache. The DMA module comprises a command acquisition DMA module and a data transmission DMA module, is controlled by the programmable state machine, and is respectively used for acquiring corresponding commands or data from the programmable state machine, and the DMA module is respectively and correspondingly provided with a command cache and a data cache for the command acquisition DMA module and the data transmission DMA module and is used for caching DMA commands and DMA data acquired in DMA operation.
According to the IO processing module provided by the embodiment of the invention, when the PCIe virtualization module is used for uniformly managing downstream PCIe devices, the functions of host IO and device IO are realized through the design of related functions of the CSR module and the DMA module, and specifically, host IO operation or device IO operation is selected according to different PCIe terminal devices corresponding to the BAR address and the functions of the PCIe terminal devices, so that the high efficiency of IO operation under different user requirements is ensured.
Based on any of the above embodiments, fig. 5 is a schematic diagram illustrating a complete structure of a system for dynamically configuring PCIe terminal devices according to an embodiment of the present invention, and the details are as follows.
The system for dynamically configuring the PCIe terminal equipment comprises a PCIe virtualization module, an external debugger and an embedded processor.
The PCIe virtualization module comprises a PCIe physical layer, a PCIe link layer, a PCIe transaction layer message distributor, a debugging interface, a configuration processing module and an IO processing module.
The configuration processing module also comprises a configuration message receiving and sending interface, a CPU configuration processing interface, a programmable hardware state machine and a state machine micro-sequence program storage unit.
The IO processing module also comprises a target IO message receiving and transmitting interface, a CSR module, a programmable state machine, a state machine micro-sequence program storage unit, a DMA module and an initial IO message receiving and transmitting interface.
The CSR module, in turn, includes a BAR address space mapping management interface, a BAR address space memory, and a BAR specific address access trigger.
The DMA module comprises a command acquisition DMA, a data transmission DMA, a command cache and a data cache; the data cache is also connected with an external data interface for writing the DMA data.
Each functional module in the system for dynamically configuring PCIe terminal device is described in the foregoing embodiments, and is not described herein again. The system for dynamically configuring the PCIe terminal equipment provided by the embodiment of the invention has the following main functions: the method comprises the steps of realizing dynamic configuration and unified management of PCIe terminal equipment through simulation of a PCIe switch; an external debugging interface, an internal embedded processor and a programmable state machine adopt a method of joint control by stages and tasks, thereby effectively meeting the requirements of comprehensively, flexibly and real-timely responding and processing PCIe transaction layer messages under the conditions of debugging, application, updating and abnormal conditions; corresponding functional design is carried out in an IO processing module in the PCIe virtualization module to support two IO operation modes of equipment IO and host IO, and the IO operation efficiency is improved.
Based on any of the above embodiments, fig. 6 is a flowchart illustrating a method for dynamically configuring a PCIe terminal device according to an embodiment of the present invention, and the specific content is as follows.
Step S610, the PCIe virtualization module of the system receives a PCIe configuration request packet sent by the host when the host scans PCIe terminal devices downstream of the system.
When the host boots up, the topology of downstream PCIe devices is scanned during the PCIe enumeration phase. When the equipment where the system is located is scanned, the PCIe virtualization module responds to be a PCIe switch; the host further scans the downstream PCIe devices of the device, and the PCIe virtualization module can replace the virtual downstream PCIe device to respond, which is represented by a logical PCIe switch and a plurality of downstream PCIe terminal devices in various logics. After the host finishes power-on startup and PCIe device scanning, the host enters a running stage, the device where the system is located can initiate a PCIe hot plug request, and multiple or multiple logical PCIe terminal devices can be deleted or added dynamically. The number and the attribute of logic PCIe terminal equipment at the downstream of the equipment of the system can be flexibly configured, and the configuration content is negotiated and determined by the configuration management server and the embedded processor of the equipment when the equipment is started or operated. In an application scenario of the data center based on the elastic bare metal server, the downstream logical PCIe terminal device may be any device type deployed on a PCI slot, such as a sound card, a network card, a built-in Modem, and the like.
Step S620, the configuration processing module of the PCIe virtualization module replies a PCIe configuration response packet to the PCIe configuration request packet, where the configuration response packet is generated by negotiation between the embedded processor of the system and an external configuration management server, and a generation sequence of the configuration response packets is determined by the embedded processor.
Specifically, in the host configuration stage, the host scans the downstream PCIe terminal equipment in the system and performs corresponding configuration work, and the PCIe virtualization module replaces the PCIe terminal equipment, receives a PCIe configuration request message sent by the host and replies a configuration response message to the configuration request message. That is, the host is implemented by the PCIe virtualization module when scanning and configuring the PCIe endpoint device. The PCIe configuration request message and the configuration response message are message types used for equipment information interaction when the PCIe configuration request message and the configuration response message are used for configuring PCIe terminal equipment in a PCIe protocol.
Step S630, the PCIe virtualization module of the system receives a PCIe IO request packet sent by the host, where the PCIe IO request packet includes a BAR address space memory read-write request corresponding to a downstream PCIe terminal device.
Step S640, the IO processing module of the PCIe virtualization module replies an IO response packet to the PCIe IO request packet, where the IO response packet includes content stored in the BAR address space memory.
In the device IO operation stage, the host may request to access the configured IO resource corresponding to the PCIe terminal device. At this time, the embodiment of the present invention is also responsible for receiving a PCIe IO request packet sent by the host through a PCIe virtualization module, and replying an IO response packet to the configuration request packet. The PCIe IO request message and the IO response message are message types used for device information interaction when accessing PCIe terminal device resources in a PCIe protocol. The PCIe IO request message comprises a BAR address space memory read-write request corresponding to downstream PCIe terminal equipment. The BAR address space is composed of a plurality of BAR addresses corresponding to a plurality of functions representing a plurality of PCIE terminal devices, and the data stored in the BAR address space memory represents base addresses of the PCIE terminal devices in the PCIE address space. Therefore, through the PCIe IO request packet including the BAR address space memory read-write request, the host may request to access the IO resource and the corresponding function corresponding to the address.
S650, when the target BAR address is accessed, triggering the programmable state machine of the IO processing module to run the BAR address space memory updating program and the DMA request program.
The embodiment of the invention embodies the application flow of the system for executing the method for dynamically configuring the PCIe terminal equipment. One possible embodiment is as follows:
a host configuration stage: in the host configuration stage, PCIe terminal equipment starts to be scanned after the host is powered on and started; after receiving the READY inquiry instruction, the terminal equipment waits for the embedded processor to be powered on; after the embedded processor is powered on and READY, waiting for the programmable state machine to return a READY signal; when the equipment of the system receives the configuration reading request, the embedded processor returns the PCIe switch type of the equipment, then the equipment receives the reading request of the downstream PCIe terminal equipment, and the equipment of the system returns the configuration information of one of the downstream equipment after communicating with the embedded processor to complete the task of the host scanning stage.
And (3) host IO stage: the embedded processor configures a CSR module in the PCIe virtualization to set BAR address information for each downstream device. Then, an IO operation phase may be performed, and when the device in which the system is located receives an IO read request, the CSR module may return CSR content of a specific downstream device. When a host accesses a particular BAR address, a BAR specific address access trigger sends a trigger signal to the programmable state machine.
And (4) equipment IO stage: after the flow of the IO stage of the equipment is triggered, a programmable state machine starts a command to obtain DMA, the command obtains DMA and sends a host memory reading command, the host returns response content to a command cache, the programmable state machine reads the command cache resolving command, starts data transmission DMA, the data transmission DMA sends a host memory reading and writing command, the host returns memory content to the data cache, and the main operation of the IO stage of the equipment is completed. A complete implementation provided above may be seen with reference to the complete flow diagram of the method for dynamically configuring a PCIe endpoint device provided in fig. 7.
Based on any of the above embodiments, fig. 8 shows a flowchart of a debugging method for PCIe terminal equipment provided in the embodiment of the present invention, and the specific content is as follows.
Step S810, starting an external debugger of the system;
step S820, the external debugger replaces the PCIe virtualization module with a debug interface provided by the PCIe virtualization module to perform PCIe configuration and IO call debugging;
step S830, after the target type PCIe terminal device is debugged, modify the program in the PCIe virtualization module for supporting the target type PCIe terminal device.
The embodiment of the invention embodies the debugging process of the system for executing the method for dynamically configuring the PCIe terminal equipment. The system for dynamically configuring the PCIe terminal equipment provided by the embodiment of the invention is connected with the PCIe virtualization module through the debugging interface provided by the PCIe virtualization module, and is used for replacing the PCIe virtualization module to debug PCIe configuration and IO call through the debugging interface provided by the PCIe virtualization module when the system is started; and after the target type PCIe terminal equipment is debugged, modifying a program in the PCIe virtualization module for supporting the target type PCIe terminal equipment.
The system for dynamically configuring the PCIe terminal equipment provided by the embodiment of the invention also has the programmable and dynamic online updating capability through the setting of the external debugger and the PCIe virtualization module, and can support the virtual realization of the existing PCIe equipment and the PCIe equipment of new types which may appear in the future. The support for different types of PCIe devices is realized by adding or modifying the program storage content of the device where the system is located, the system can simultaneously support a plurality of different types of PCIe devices, and the types of the support devices are limited by the program storage capacity. Aiming at the new PCIe equipment which is not realized in the system, the system provides an external debugger, a corresponding debugging interface is arranged in a PCIe virtualization module to be connected with the external debugger 120, the external debugger can take over the function of processing configuration/IO messages of the PCIe virtualization module when the debugging function is operated, and then the external debugger host replaces the PCIe virtualization module to complete the function simulation of a PCIe switch and downstream PCIe terminal equipment. After the PCIe terminal device of the new type completes all functions in the external host debugging mode, the corresponding program storage content in the PCIe virtualization module can be modified, and the support for the PCIe terminal device of the new type is updated.
In the system, the external debugger has the highest priority, so that whether the external debugger is connected or not can be judged after the equipment where the system is located is powered on. If so, the system enters a debug state. When the external debugger is started to carry out debugging, the external debugger can replace the PCIe virtualization module to carry out debugging of PCIe configuration and IO calling through a debugging interface provided by the PCIe virtualization module, and modification of existing PCIe equipment and updating of new PCIe equipment are achieved. And after debugging is finished, modifying corresponding program storage content in the PCIe virtualization module in a programmable mode, thereby realizing the support of the new type of PCIe terminal equipment.
Through the process, the system realizes the dynamic update of the number or the attributes of the PCIe terminal devices. When the PCIe terminal equipment needs to be updated, an external debugger is started to intervene, the PCIe terminal equipment can be dynamically configured, and meanwhile, the problem that the configuration flexibility is poor when the traditional SR-IOV method is used for multiplexing the PCIe terminal equipment is solved.
Based on any of the above embodiments, fig. 9 is a schematic flow chart illustrating an exception handling method for dynamically configuring PCIe terminal device according to an embodiment of the present invention, and the specific content is as follows.
Step S910, a programmable state machine of a configuration processing module of the PCIe virtualization module replaces a negotiation process of the embedded processor and an external configuration management server to generate the configuration response message;
step S920, if the PCIe configuration request packet is of an undefined type, the configuration response packet is generated instead of the programmable state machine according to negotiation between the embedded processor and an external configuration management server.
The embodiment of the invention embodies the exception handling process of the system for executing the method for dynamically configuring the PCIe terminal equipment. The programmable hardware state machine of the configuration processing module in the embodiment of the invention also has the function of processing PCIe configuration transaction request messages. The programmable hardware state machine can start working after being electrified, can analyze the PCIe configuration transaction request message according to the pre-programmed configuration transaction response message, and can respond the most basic PCIe protocol, control the response of the PCIe protocol, suspend PCIe protocol interaction and the like. Meanwhile, the programmable hardware state machine can work together with the CPU configuration processing interface after the CPU configuration processing interface starts to work after being electrified.
Specifically, in the embodiment of the present invention, the configuration message may be separately received and sent by the CPU configuration processing interface or the programmable hardware state machine, or may be completed by the cooperation of the two. One possible way of cooperation is: the programmable hardware state machine is started immediately after being electrified, and after receiving the configuration read-write request of the upstream port, the programmable hardware state machine sends a pre-programmed message until the CPU completes the electrification initialization, and the CPU takes over the programmable state machine to complete the subsequent configuration read-write request response. Another possible way of cooperation is: the programmable state machine completes all preset read-write request responses, when an abnormal condition except the preset condition is met, the embedded processor is triggered to take over processing, and after the processing is completed, the programmable state machine still completes the request responses.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A system for dynamically configuring a PCIe endpoint device, the system comprising:
the PCIe virtualization module is used for receiving a PCIe configuration request message sent by a host when the host scans PCIe terminal equipment at the downstream of the system and replying a configuration response message to the PCIe configuration request message; the PCIe IO request message is used for receiving a PCIe IO request message sent by the host, and an IO response message is replied to the PCIe IO request message; the PCIe IO request message comprises a reading and writing request of a base address register BAR address space memory corresponding to downstream PCIe terminal equipment;
the external debugger is connected with the PCIe virtualization module through a debugging interface provided by the PCIe virtualization module and used for replacing the PCIe virtualization module to debug PCIe configuration and IO call through the debugging interface provided by the PCIe virtualization module during starting; the system comprises a PCIe virtualization module, a target type PCIe terminal device and a program, wherein the PCIe virtualization module is used for modifying the program used for supporting the target type PCIe terminal device in the PCIe virtualization module after the target type PCIe terminal device is debugged;
and the embedded processor is used for negotiating with an external configuration management server to generate the configuration response message and determining the generation sequence of the plurality of configuration response messages.
2. The system for dynamically configuring a PCIe endpoint device as recited in claim 1, wherein the PCIe virtualization module comprises:
the PCIe transaction layer message distributor is used for analyzing PCIe transaction layer messages received from a PCIe link layer and respectively sending the PCIe configuration request messages or the PCIe IO request messages to corresponding configuration processing modules or IO processing modules according to PCIe configuration request messages or PCIe IO request messages indicated by message header information of the PCIe transaction layer messages; the PCIe transaction layer message is used for distributing the PCIe transaction layer message received from the configuration processing module or the IO processing module to a PCIe link layer;
the debugging interface is connected with the external debugger and used for sending and receiving PCIe transaction layer messages by replacing the PCIe transaction layer message distributor when the external debugger is started;
the configuration processing module is used for receiving a PCIe configuration request message and finishing updating of PCIe terminal equipment according to the PCIe configuration request message;
and the IO processing module is used for receiving a PCIe IO request message and controlling the PCIe virtualization module and/or at least one PCIe terminal device to access a control state register and a direct memory according to the PCIe IO request message.
3. The system for dynamically configuring a PCIe endpoint device according to claim 2, wherein the configuration processing module comprises:
the configuration message receiving and sending interface is used for receiving PCIe configuration request messages distributed by the PCIe transaction layer message distributor and sending configuration response messages received from the programmable hardware state machine or the CPU configuration processing interface;
the CPU configuration processing interface is used for receiving PCIe configuration request messages forwarded by the configuration message transceiving interface and sending configuration response messages received from the embedded processor;
the programmable hardware state machine is used for receiving and analyzing the PCIe configuration request message forwarded by the configuration message transceiving interface and sending a pre-programmed configuration response message according to the analysis content;
and the state machine micro-sequence program storage unit is used for storing instructions and data of the programmable hardware state machine, and the instructions and the data are written in or read out from the CPU configuration processing interface.
4. The system for dynamically configuring a PCIe endpoint device according to claim 2, wherein the IO processing module comprises:
the target IO message receiving and sending interface is used for receiving PCIe IO request messages forwarded by the PCIe transaction layer message distributor and analyzing the PCIe IO request messages into BAR address space memory read-write requests; the memory read-write response returned by the CSR module is encapsulated into an IO response message and sent to a PCIe transaction layer message distributor;
and the CSR module is used for receiving the BAR address space memory read-write request sent by the target IO message transceiving interface, acquiring corresponding read-write request content according to the BAR address space memory read-write request and sending the read-write request content to the target IO message transceiving interface.
5. The system for dynamically configuring PCIe end devices according to claim 4, wherein the CSR module comprises:
the BAR address space mapping management interface is used for providing a BAR address corresponding to a target function of corresponding PCIe terminal equipment after receiving a BAR address space memory read-write request;
and the BAR address space memory is used for storing and providing the content of the BAR address space.
6. The system for dynamically configuring PCIe endpoint devices according to claim 5, wherein the CSR module further comprises a BAR specific address access trigger for triggering the programmable state machine of the IO processing module to run a BAR address space memory update procedure and a direct register access (DMA) request procedure when the target BAR address is accessed;
correspondingly, the IO processing module further includes:
the programmable state machine is used for responding to a BAR specific address access trigger signal sent by the BAR specific address access trigger and sending the update information of the address space memory of the BAR to the CSR module; the DMA module is also used for sending a DMA request to the DMA module and receiving a completion response of the DMA module;
the starting IO message transceiving interface is used for receiving a host memory address space memory read-write request sent by the DMA module, packaging the host memory address space memory read-write request into a PCIe IO request message and sending the PCIe IO request message to a PCIe transaction layer message distributor; and is used for analyzing the IO response message returned by the PCIe transaction layer message distributor into a host memory address space memory read-write response and sending the host memory address space memory read-write response to the DMA module;
the DMA module is used for responding to a DMA request of the programmable state machine, initiating a command to acquire DMA or data transmission DMA operation, and sending a read-write request of a memory address space memory of the host to the initial IO message transceiving interface; and writing the read-write response content of the memory in the memory address space of the host computer received from the initial IO message transceiving interface into a command cache or a data cache.
7. A method for dynamically configuring PCIe terminal equipment, which is applied to the system of any one of claims 1-6, and is characterized in that the method comprises the following steps:
a PCIe virtualization module of the system receives a PCIe configuration request message sent by a host when the host scans PCIe terminal equipment at the downstream of the system;
a configuration processing module of the PCIe virtualization module replies a PCIe configuration response message to the PCIe configuration request message, wherein the configuration response message is generated by negotiation between an embedded processor of the system and an external configuration management server, and the generation sequence of the configuration response messages is determined by the embedded processor;
a PCIe virtualization module of the system receives a PCIe IO request message sent by the host, wherein the PCIe IO request message comprises a BAR address space memory read-write request corresponding to downstream PCIe terminal equipment;
and an IO processing module of the PCIe virtualization module replies an IO response message to the PCIe IO request message, wherein the IO response message comprises the content stored in the BAR address space memory.
8. The method for dynamically configuring a PCIe endpoint device as recited in claim 7, further comprising:
when the target BAR address is accessed, the programmable state machine of the IO processing module is triggered to run a BAR address space memory update program and a DMA request program.
9. The method for dynamically configuring a PCIe endpoint device as recited in claim 7, further comprising:
starting an external debugger of the system;
the external debugger replaces the PCIe virtualization module to debug PCIe configuration and IO call through a debugging interface provided by the PCIe virtualization module;
and after the target type PCIe terminal equipment is debugged, modifying a program which is used for supporting the target type PCIe terminal equipment in the PCIe virtualization module.
10. The method for dynamically configuring a PCIe endpoint device as recited in claim 7, further comprising:
replacing a negotiation process of the embedded processor and an external configuration management server by a programmable hardware state machine of a configuration processing module of the PCIe virtualization module to generate the configuration response message;
and if the PCIe configuration request message is of an undefined type, generating the configuration response message instead of the programmable state machine according to the negotiation between the embedded processor and an external configuration management server.
CN202111140694.5A 2021-09-28 2021-09-28 System and method for dynamically configuring PCIe terminal equipment Active CN113904938B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111140694.5A CN113904938B (en) 2021-09-28 2021-09-28 System and method for dynamically configuring PCIe terminal equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111140694.5A CN113904938B (en) 2021-09-28 2021-09-28 System and method for dynamically configuring PCIe terminal equipment

Publications (2)

Publication Number Publication Date
CN113904938A CN113904938A (en) 2022-01-07
CN113904938B true CN113904938B (en) 2022-04-29

Family

ID=79029721

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111140694.5A Active CN113904938B (en) 2021-09-28 2021-09-28 System and method for dynamically configuring PCIe terminal equipment

Country Status (1)

Country Link
CN (1) CN113904938B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115632976A (en) * 2022-10-25 2023-01-20 深圳市楠菲微电子有限公司 PCIE transaction layer message generation method, device and storage medium
CN115529275B (en) * 2022-11-28 2023-04-07 中国人民解放军国防科技大学 Message processing system and method
CN115857995B (en) * 2023-02-08 2023-05-16 珠海星云智联科技有限公司 Method, medium and computing device for upgrading interconnection device
CN115904520B (en) * 2023-03-13 2023-06-06 珠海星云智联科技有限公司 Configuration preservation method based on PCIE topology state change and related equipment
CN116521596B (en) * 2023-06-29 2023-09-22 北京大禹智芯科技有限公司 PCIe Switch simulator realization method and device based on Qemu virtual environment
CN116501666B (en) * 2023-06-29 2023-09-01 北京大禹智芯科技有限公司 Implementation method of PCIe Switch simulator
CN117743212A (en) * 2024-02-19 2024-03-22 北京光润通科技发展有限公司 PCIe IP card verification method and system for PCIe device with multiple physical functions based on FPGA

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9996484B1 (en) * 2014-09-17 2018-06-12 Amazon Technologies, Inc. Hardware acceleration for software emulation of PCI express compliant devices
CN110489365A (en) * 2016-01-13 2019-11-22 华为技术有限公司 Switching equipment, peripheral parts interconnected High Speed System and its initial method
CN110659245A (en) * 2018-06-29 2020-01-07 英特尔公司 System with programmable multi-context accelerator circuitry
CN112559120A (en) * 2019-09-26 2021-03-26 无锡江南计算技术研究所 Customized PCIE bus IO virtualization supporting method
US11106616B1 (en) * 2019-11-21 2021-08-31 Xilinx, Inc. Virtualized peripheral component interconnect express (PCIe) device

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102707991B (en) * 2012-05-17 2016-03-30 中国科学院计算技术研究所 The many virtual shared method and systems of I/O
BR112018000362A2 (en) * 2016-11-09 2018-09-11 Huawei Technologies Co., Ltd. packet processing method in cloud computing system, host and system
CN113312143B (en) * 2021-03-03 2024-01-23 阿里巴巴新加坡控股有限公司 Cloud computing system, command processing method and virtualization simulation device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9996484B1 (en) * 2014-09-17 2018-06-12 Amazon Technologies, Inc. Hardware acceleration for software emulation of PCI express compliant devices
CN110489365A (en) * 2016-01-13 2019-11-22 华为技术有限公司 Switching equipment, peripheral parts interconnected High Speed System and its initial method
CN110659245A (en) * 2018-06-29 2020-01-07 英特尔公司 System with programmable multi-context accelerator circuitry
CN112559120A (en) * 2019-09-26 2021-03-26 无锡江南计算技术研究所 Customized PCIE bus IO virtualization supporting method
US11106616B1 (en) * 2019-11-21 2021-08-31 Xilinx, Inc. Virtualized peripheral component interconnect express (PCIe) device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于PCIe的多路传输系统的DMA控制器设计;李胜蓝等;《计算机应用》;20170310(第03期);参见全文 *

Also Published As

Publication number Publication date
CN113904938A (en) 2022-01-07

Similar Documents

Publication Publication Date Title
CN113904938B (en) System and method for dynamically configuring PCIe terminal equipment
KR102428317B1 (en) A novel ssd architecture for fpga based acceleration
JP3601955B2 (en) Data transfer method and computer system suitable for it
US20170329625A1 (en) Method, apparatus, and system for accessing storage device
US5721876A (en) Sockets application program mechanism for proprietary based application programs running in an emulation environment
US20110069710A1 (en) Switching Method
CN101751371A (en) Chipset support for non-uniform memory access among heterogeneous processing units
CN106662895B (en) The method of computer equipment and computer equipment reading and writing data
CN115080479B (en) Transmission method, server, device, bare metal instance and baseboard management controller
CN116541227A (en) Fault diagnosis method and device, storage medium, electronic device and BMC chip
CN115658586A (en) Resource management chip, resource management method, electronic device, and readable storage medium
US7996524B2 (en) Controlling external communication of embedded device using proxy server
CN116521324B (en) Interrupt virtualization processing method and device and electronic equipment
CN109101456B (en) Data interactive communication method and device in simulation SSD and terminal
US7930445B2 (en) Computer system using remote I/O and I/O data transfer method
CN108509155B (en) Method and device for remotely accessing disk
US7827194B2 (en) Access to shared disk device on storage area network
JP6965943B2 (en) Configuration management device, configuration management system, configuration management method, and configuration management program
JP5975003B2 (en) Virtualization control device, virtualization system, virtualization method, and virtualization control program.
CN109117001B (en) Android system gyroscope event conversion method and device
JP3800305B2 (en) Gateway system and communication device terminal
CN117519955B (en) Control method, system and device of acceleration equipment and acceleration equipment
JP2004094800A (en) Program simulation device and its method
JPH0736374A (en) Data link system
CN115237626A (en) Data processing method and device, electronic equipment and computer readable medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant