Embodiment
Intel Virtualization Technology is a kind of by the decoupling method of bottom hardware equipment and upper strata operating system, separate, it introduces monitor of virtual machine (Virtual Machine Monitor, being called for short VMM) layer directly manages bottom hardware resource, and the virtual machine that establishment is irrelevant with bottom hardware (Virtual Machine is called for short VM) keeps supplying layer operation system and application program uses.Virtual machine does not have real physical equipment, needs, with software simulation interruptable controller and physical equipment, to form virtual interrupt controller and virtual unit.Before os starting, subjunctive BIOS writes ACPI table in the internal memory distributing to affiliated virtual machine.After os starting, can be arranged by instruction and use virtual unit.VMM, after these instructions intercepting operating system, arranges virtual unit or operating physical equipment in analog according to the instruction of operating system, and analog result is returned to virtual machine.
Distributed virtualization technology is a kind of by the resource from multiple stage physical machine, be reassembled into the technology of a virtual machine in a virtual manner, namely the virtual hardware of distributed virtual machine is distributed in different node, in order that use the hardware device on different node.A kind of distributed virtualization structure that Fig. 2 provides for the embodiment of the present invention.Distributed virtualization framework as shown in Figure 2 comprises VM layer, DVMM layer and physical node layer.Each physical node comprises the hardware resources such as processor, internal memory, disk, network interface card and peripheral hardware.Each physical node is all deployed with a DVMM, and when creating a virtual machine, the DVMM be deployed on each physical node can at the virtual hardware equipment of physical node needed for virtual machine creating affiliated separately.Virtual machine, in section at the same time, runs based on a physical node.DVMM mainly comprises that CPU is virtual, internal memory virtualization and the three large modules such as I/O is virtual, has reached the physical resource polymerization being polymerized to approach with virtual resource.Wherein I/O is virtual can simulate the PCI equipment under non-virtualized environment and interrupt syndeton.When os starting being on a virtual machine installed and running when initialization and use hardware resource, DVMM can not allow all these action direct controls on physical hardware, but responsive access behavior can be intercepted and captured, and this access behavior is modeled to the access to virtual hardware, final is that agency goes to operate real physical hardware with virtual hardware, ensure that the correctness that isolation between different virtual machine and virtual machine use hardware resource.DVMM layer directly manages local bottom hardware resource, and is mutually cooperated with the DVMM on other nodes by the mode of network service, makes virtual machine have the ability of cross-node access resources.Particularly, after the distributed virtual machine monitor of a node intercepts and captures the instruction of VME operating system, first to judge virtual hardware (comprising the pin of virtual interrupt controller) operated by this instruction is at which node, then the operation that will carry out is sent to this node by communication module and goes simulation, such as, send to virtual hardware is set or real physical equipment corresponding to operation virtual hardware goes simulation.
The virtual interrupt pin management method process flow diagram of a kind of distributed virtualization system that Fig. 3 provides for the embodiment of the present invention.The virtual PCI device that Fig. 4 provides for the embodiment of the present invention interrupts link composition.As shown in Figure 4, the virtual interrupt controller that VM provides is overall vIO-APIC, interrupt pin on overall situation vIO-APIC is called overall virtual interrupt pin, the overall virtual interrupt controller that virtual OS only has VM to provide to it, the interrupt pin of each virtual PCI device is connected on an overall virtual interrupt pin on overall virtual interrupt controller.The virtual interrupt controller that DVMM on each physical node provides is local vIO-APIC, interrupt pin on the vIO-APIC of local is called local virtual interrupt pin, simulate overall virtual interrupt pin by the local virtual interruptable controller on each physical node, the interrupt pin of each virtual PCI device is connected to a local virtual interrupt pin on local virtual interruptable controller.A physical node has multiple virtual PCI device, and a virtual PCI device can have multiple interrupt pin, and virtual PCI device also can be connected to the multiple local virtual interrupt pin on the vIO-APIC of local.Pci bus belonging to the virtual PCI device registration that DM module creates in for each physical node number and device number, the device number Unified number of the virtual PCI device on all physical nodes, the local virtual interrupt pin on the local vIO-APIC on each physical node can be numbered separately.The interrupt structure that VM provides to VME operating system is, the annexation of the overall virtual interrupt pin on the virtual interrupt pin of the virtual PCI device that this VM comprises and overall vIO-APIC.And the interrupt structure being deployed in the DVMM on a physical node and providing is, the annexation of the local virtual interrupt pin on the local vIO-APIC that the interrupt pin of the virtual PCI device that this physical node comprises and this physical node manage.Such as, in the interrupt structure that virtualizing operating systems is known, device number is that the Intx interrupt pin of the virtual PCI device of Device is connected on the overall virtual interrupt pin of in overall vIO-APIC No. gsi; And to be the interrupt structure that the physical node at the virtual PCI device place of Device is preserved at device number be, device number is that the Intx interrupt pin of the virtual PCI device of Device is connected in the local virtual interrupt pin of xth number in the vIO-APIC of local, namely, on physical node, simulate the overall virtual interrupt pin of in overall vIO-APIC No. gsi by the local virtual interrupt pin of xth in the vIO-APIC of local number.When VME operating system will go to arrange certain overall virtual interrupt pin, DVMM can intercept and capture this operation, and be modeled to this overall virtual interrupt pin is set in the physical node of specifying, described physical node of specifying is the local virtual interrupt pin place physical node being connected to this overall virtual interrupt pin.
As shown in Figure 3, the method that the present embodiment provides comprises:
Step 31: the subjunctive BIOS module of virtual machine sets up the annexation of the interrupt pin of overall virtual interrupt pin and virtual PCI device in ACPI table, wherein, described overall virtual interrupt pin is the one or more interrupt pin on virtual interrupt pin controller, and any one in described overall virtual interrupt pin is corresponding to the interrupt pin of one or more described virtual PCI device belonging to same physical node; Described virtual interrupt pin controller is for the operating system of described virtual machine provides;
Described ACPI can be write the virtual memory of virtual machine by the subjunctive BIOS module of distributed virtualization system.After ACPI table write virtual memory, VME operating system can obtain ACPI table from internal memory, can know on which the virtual interrupt pin which virtual PCI device which overall virtual interrupt pin is connected on according to ACPI table.In all annexations of ACPI table record, any one overall virtual interrupt pin is connected to the interrupt pin of the one or more virtual PIC equipment on same physical node.That is, the virtual PCI device on different physical node can not be connected on an identical overall virtual interrupt pin.Therefore, achieve same overall virtual interrupt pin and go simulation by the local virtual interrupt pin on a physical node.Such as, be that the Intx interrupt pin of the virtual PCI device of Device is connected on the overall virtual interrupt pin of in overall vIO-APIC No. gsi by device number, show above-mentioned annexation at ACPI and can be expressed as (virtual PCI device Device, interrupt pin Intx, overall virtual interrupt pin number gsi).
As shown in Figure 4, No. 10 overall virtual interrupt pin of overall situation vIO-APIC is connected to the Intx3 interrupt pin of virtual PCI device vDEV12 on the Intx1 interrupt pin of virtual PCI device vDEV11 on NODE1 node and NODE1 node, and No. 20 overall virtual interrupt pin of overall vIO-APIC is connected to the Intx1 interrupt pin of virtual PCI device vDEV22 on the Intx1 interrupt pin of virtual PCI device vDEV21 on NODE2 node and NODE2 node.Can know from above-mentioned annexation, No. 10 overall virtual interrupt pin of overall vIO-APIC is connected to the virtual PCI device on NODE1 node, and No. 20 overall virtual interrupt pin of overall vIO-APIC is connected to the virtual PCI device on NODE2 node.
The method of the interrupt pin on the virtual PCI device making an overall virtual interrupt pin be connected on same physical node has multiple.Preferably, all overall virtual interrupt pins are divided into multiple mutually disjoint subset by subjunctive BIOS module, during local virtual interrupt pin on mapping physical node, the interrupt pin of all PCI equipment of same physical node is mapped in a subset.Therefore, in ACPI table, the interrupt pin of all PCI equipment of a physical node is connected in a subset.Such as, on No. 0 physical node, the device number of virtual PCI device is between 0 ~ 10, on No. 1 physical node, the device number of virtual PCI device is between 11 ~ 13, on No. 2 physical nodes, the device number of virtual PCI device is between 14 ~ 16, ..., on No. 7 nodes, the device number of virtual PCI device is between 29 ~ 31, and the interruption stitch (i.e. all PCI device interrupt stitch of No. 0 node) of 0 ~ No. 10 PCI equipment can be direct-connected to 16 ~ No. 26 vIO-APIC stitch; The interruption stitch (i.e. all PCI device interrupt stitch of No. 1 node) of 11 ~ No. 13 PCI equipment can be direct-connected to 27 ~ No. 29 vIO-APIC stitch; The interruption stitch (i.e. all PCI device interrupt stitch of No. 7 nodes) of 29 ~ No. 31 PCI equipment can be direct-connected to 45 ~ No. 47 vIO-APIC stitch.0 ~ No. 15 stitch of vIO-APIC is as specific use.If all overall virtual interrupt pins are divided into multiple mutually disjoint subset, after the interrupt pin of all PCI equipment of physical node correspondence is in a subset, operating system once can arrange all overall virtual interrupt pin in this subset, and do not need to arrange these overall virtual interrupt pins several times, because the local virtual interrupt pin simulating these overall virtual interrupt pins is on same physical node.Thus, the performance of virtual machine can be improved.
Alternatively, the interrupt pin of all virtual PCI device on a physical node also can be all connected to same overall virtual interrupt pin by subjunctive BIOS module, the corresponding overall virtual interrupt pin of all local virtual interrupt pin namely in ACPI table on a physical node.
Step 32: the operating system of virtual machine obtains described ACPI and shows, and shows according to described ACPI, operates to the instruction of the overall virtual interrupt pin Offered target vCPU in described ACPI table.
The virtual PCI device that the operating system of virtual machine connects according to overall virtual interrupt pin in APCI table, determine to process the destination virtual central processing unit (VirtualCentral Processing Unit is called for short vCPU) that each overall virtual interrupt pin interrupts.Such as, the virtual PCI device that a global interrupt pin in APCI table connects is virtual mouse, the target vCPU of the interruption of this virtual PCI device of process can be set to multiple vCPU by VME operating system, and the virtual PCI device that another global interrupt pin in APCI table connects is virtual hard disk, the target vCPU of the interruption of this virtual PCI device of process can be set to a vCPU by VME operating system.During the vCPU that the overall virtual interrupt pin of VME operating system set handling one interrupts, send to overall virtual interrupt controller and arrange instruction, this arranges instruction and by which vCPU is processed for arranging the interruption that overall virtual interrupt pin produces.
It should be noted that, each overall virtual interrupt pin in APCI table needs Offered target vCPU, can be process one by one, also can be batch processing, in following examples, for the ease of understanding, be specifically that one of them overall virtual interrupt pin arranges vCPU to illustrate for single, those skilled in the art are appreciated that in each step of following examples completely, also can carry out identical process for the whole overall virtual interrupt pin in APCI table simultaneously.
Step 33: the distributed virtual machine monitor intercepting described instruction, the physical node at the local virtual interrupt pin place that described overall virtual interrupt pin is corresponding is determined according to distributed interrupt pinout information, described local virtual interrupt pin to be positioned on physical node and for simulating described overall virtual interrupt pin, described distributed interrupt pinout information comprises the annexation of the interrupt pin of overall virtual interrupt pin and virtual PCI device, and the virtual PCI device that comprises of each physical node and local virtual interrupt pin.
Following distributed interrupt pinout information can be recorded: Yi Zhongshi in the storage system of distributed virtualization system, the annexation of the interrupt pin of virtual PCI device on each overall virtual interrupt pin and physical node, this annexation show with ACPI in annexation identical.Another kind is, the virtual PCI device that each physical node comprises and local virtual interrupt pin.Distributed interrupt pinout information is not likely stored in intercepts and captures the physical node belonging to DVMM that VME operating system arranges instruction, intercept and capture VME operating system and the DVMM of instruction is set by mutually cooperating between DVMM, obtain distributed interrupt pinout information from other physical node.
In physical node, the local virtual interrupt pin that the interrupt pin of virtual PCI device connects, the overall virtual interrupt pin that the interrupt pin that physical node is simulated this virtual PCI device connects.As shown in Figure 4, be deployed in the interrupt structure that on NODE1 node, DVMM provides, on NODE1 node, the Intx1 interrupt pin of virtual PCI device vDEV11 is connected to No. 5 local virtual interrupt pin of the local vIO-APIC on NODE1 node, and on NODE1 node, the Intx3 interrupt pin of virtual PCI device vDEV12 is connected to No. 6 local virtual interrupt pin of the local vIO-APIC on NODE1 node.As shown in Figure 4, be deployed in the interrupt structure that on NODE2 node, DVMM provides, on NODE2 node, the Intx1 interrupt pin of virtual PCI device vDEV21 is connected to No. 9 local virtual interrupt pin of the local vIO-APIC on NODE2 node, and on NODE2 node, the Intx1 interrupt pin of virtual PCI device vDEV22 is connected to No. 10 local virtual interrupt pin of the local vIO-APIC on NODE2 node.
When VME operating system is the target vCPU of the interruption that it produces an overall virtual interrupt pin set handling, run VME operating system physical node deploy DVMM intercept and capture VME operating system arrange global interrupt virtual pin instruction is set after, according to the annexation of the interrupt pin of virtual PCI device on each overall virtual interrupt pin recorded and physical node, determine the interrupt pin of the virtual PCI device corresponding to overall virtual interrupt pin be set up.The local virtual interrupt pin that the virtual PCI device comprised by each physical node of record and each physical node are responsible for, can determine the physical node at the local virtual interrupt pin place of simulating.
Step 34: the DVMM on the physical node at described local virtual interrupt pin place preserves the corresponding relation of described local virtual interrupt pin and described target vCPU according to described instruction, described target vCPU is for the treatment of the interruption in described local virtual interrupt pin.
When the local physical node of the physical node at described local virtual interrupt pin place and the DVMM of capturing operation Operation system setting instruction is Same Physical node, the DVMM of capturing operation Operation system setting instruction preserves described local virtual interrupt pin and the corresponding relation processing the target vCPU that described local virtual interrupt pin is interrupted, such as be kept in interrupt processing vCPU table, processing the target vCPU that described local virtual interrupt pin is interrupted is the target vCPU arranged in the instruction of operating system.Have no progeny in the interrupt pin being connected to the virtual PCI device in described local virtual interrupt pin produces, which vCPU the DVMM on this local virtual interrupt pin place physical node can determine by according to the interrupt processing vCPU table preserved to process the interruption in this local virtual interrupt pin.When the local physical node at the physical node at described local virtual interrupt pin place and the DVMM place of capturing operation Operation system setting instruction is not Same Physical node, the DVMM of capturing operation Operation system setting instruction sends to the far-end physical node at described local virtual interrupt pin place by arranging instruction, the corresponding relation of described local virtual interrupt pin and target vCPU is preserved by the DVMM of far-end physical node, such as be kept in interrupt processing vCPU table, thus be connected to the virtual PCI device in described local virtual interrupt pin produce in have no progeny, which vCPU the DVMM of far-end physical node can determine by according to the interrupt processing vCPU table preserved to process the interruption in described local virtual interrupt pin.
The method that the present embodiment provides, an overall virtual interrupt pin in subjunctive BIOS module write ACPI table is connected in the interrupt pin of one or more virtual PIC equipment on same physical node, namely an overall virtual interrupt pin goes simulation by the local virtual interrupt pin on same physical node, interrupt pin due to the virtual PCI device of different physical nodes can not be connected on same overall virtual interrupt pin, VME operating system is when arranging this overall virtual interrupt pin, do not need to the packet of different physical nodes transmission for synchronously arranging, improve the performance of distributed virtualization system.
Further, if the DVMM of the interrupt pin of local virtual described in step 35 place physical node, determining to process according to the instruction of operating system the target vCPU that described local virtual interrupt pin is interrupted is vCPU on far-end physical node, the DVMM of this local virtual interrupt pin place physical node receives in this local virtual interrupt pin and has no progeny, interrupt processing vCPU according to preserving shows, the target vCPU of this interrupt delivery to far-end physical node to be processed, cause the processing time longer, the mode that this non-localized process is interrupted can produce larger impact to the performance of virtual machine.For avoidance breakout is by the phenomenon of non-local vCPU process, the embodiment of the present invention additionally provides following methods and realizes interrupting localization process: as shown in Figure 5, step 34 can comprise:
Whether step 340: the DVMM of local virtual interrupt pin place physical node is local physical node according to described instruction determination target vCPU place physical node.If perform step 341, otherwise perform step 342.
Step 341: the corresponding relation preserving target vCPU in local virtual interrupt pin and described instruction.
Step 342: if local physical node comprises more than one vCPU, the target vCPU arranged in instruction is revised as the vCPU on local physical node, the corresponding relation of target vCPU after preserving local virtual interrupt pin and revising.
If the DVMM of local virtual interrupt pin place physical node, judge that the target vCPU place physical node processing the interruption that the global interrupt pin that is set up produces is not local physical node according to arranging instruction, and local physical node comprises multiple vCPU, can according to the load state of local vCPU, the target vCPU arranging in instruction the interruption that the global interrupt pin that processes and be set up produces is revised as the vCPU on local physical node.Have no progeny during the local virtual interrupt pin simulating this overall virtual interrupt pin produces, this interrupt delivery processes to the vCPU on local physical node by local DVMM, this interrupt delivery is not needed to process to far-end physical node, alleviate internodal pressure communication, reduce the impact interrupted virtual machine performance, improve the performance of virtual machine in interrupt procedure.
The virtual interrupt pin management method process flow diagram of another distributed virtualization system that Fig. 6 provides for the embodiment of the present invention.The present embodiment be to the further refinement of the corresponding embodiment of Fig. 5 after embodiment.As shown in Figure 6, the method that the present embodiment provides comprises:
Step 60: the subjunctive BIOS module of virtual machine sets up the annexation of the interrupt pin of overall virtual interrupt pin and virtual PCI device in ACPI table, wherein, described overall virtual interrupt pin is the one or more interrupt pin on virtual interrupt pin controller, and any one in described overall virtual interrupt pin is corresponding to the interrupt pin of one or more described virtual PCI device belonging to same physical node; Described virtual interrupt pin controller is for the operating system of described virtual machine provides;
Step 61: the operating system of virtual machine obtains described ACPI and shows, and shows according to described ACPI, operates to the instruction of the overall virtual interrupt pin Offered target vCPU in described ACPI table.
Step 62: the DVMM intercepting and capturing described instruction, according to distributed virtual interrupting information, determines the local virtual interrupt pin of the overall virtual interrupt pin that described simulation is set up.
Step 63: the DVMM intercepting and capturing described instruction, according to distributed virtual interrupting information, determines that whether the physical node at described local virtual interrupt pin place is the local physical node at this DVMM place.
If the physical node at described local virtual interrupt pin place is local physical node, the DVMM of capturing operation system directive performs step 641, step 642 and step 643, otherwise the DVMM of capturing operation system directive performs step 65.
Step 641: judge according to described instruction whether the target vCPU place physical node processing the interruption that the global interrupt pin that is set up produces is local physical node.If perform step 642, otherwise perform step 643.
Step 642: the target vCPU place physical node processing the interruption that the global interrupt pin that is set up produces is local physical node, according to the corresponding relation arranging instruction and preserve described local virtual interrupt pin and target vCPU.
Step 643: the target vCPU place physical node of the interruption that the global interrupt pin that process is set up produces is not local physical node and local physical node comprises more than one vCPU, the target vCPU of the interruption that global interrupt pin process be set up produced is revised as the target vCPU on local physical node, the corresponding relation of target vCPU after preserving described local virtual interrupt pin and revising.
Step 65: described instruction is sent to the far-end physical node at described local virtual interrupt pin place by the DVMM intercepting and capturing described instruction.The far-end physical node receiving instruction performs step 641, step 642 and step 643.
The virtual interrupt management devices structural representation of a kind of distributed virtualization system that Fig. 7 provides for the embodiment of the present invention.As shown in Figure 7, this device comprises: virtual machine and multiple distributed virtual machine monitor be deployed on different physical node.Wherein, virtual machine comprises subjunctive BIOS module 70 and operating system 71, and distributed virtual machine monitor can be the first distributed virtual machine monitor 72 and the second distributed virtual machine monitor 73.
Described subjunctive BIOS module 70, for setting up the annexation of the interrupt pin of overall virtual interrupt pin and virtual PCI device on virtual machine in ACPI table, any one in wherein said overall virtual interrupt pin is corresponding to the interrupt pin of one or more described virtual PCI device belonging to same described physical node.Subjunctive BIOS module 70 can by the virtual memory of described ACPI table write virtual machine.Preferably, described subjunctive BIOS module is specifically for being divided into multiple mutually disjoint subset by described overall virtual interrupt pin, set up the annexation of the interrupt pin of described overall virtual interrupt pin and virtual PCI device, wherein, the interrupt pin belonging to all virtual PCI device of same described physical node is connected in same described subset.Or, described subjunctive BIOS module is specifically for setting up the annexation of the interrupt pin of described overall virtual interrupt pin and virtual PCI device, and all virtual PCI device interrupt pin wherein on same physical node are corresponding to same overall virtual interrupt pin.
Operating system 71, showing for obtaining described ACPI, showing according to described ACPI, operates to the instruction of the overall virtual interrupt pin Offered target vCPU in described ACPI table.
The first distributed virtual machine monitor 72 in described multiple distributed virtual machine monitor, for intercepting and capturing the instruction of described operating system Offered target vCPU, the physical node at the local virtual interrupt pin place that described overall virtual interrupt pin is corresponding is determined according to distributed interrupt pinout information, described local virtual interrupt pin to be positioned on described physical node and for simulating described overall virtual interrupt pin, described distributed interrupt pinout information comprises the annexation of the interrupt pin of virtual PCI device on overall virtual interrupt pin and physical node on described virtual machine, and the local virtual interrupt pin that the virtual PCI device that comprises of each physical node and each physical node described are responsible for.
Further, first distributed virtual machine monitor 72, time also for determining that the physical node at described local virtual interrupt pin place is not local physical node, described instruction is sent to the distributed virtual machine monitor on the physical node of described local virtual interrupt pin place.
The second distributed virtual machine monitor 73 in described multiple distributed virtual machine monitor, for preserving the corresponding relation of described local virtual interrupt pin and described target vCPU according to described instruction, described second distributed virtual machine monitor is the distributed virtual machine monitor on the physical node at described local virtual interrupt pin place.
Further, second distributed virtual machine monitor 73, have no progeny in also producing in the interrupt pin being connected to the virtual PCI device in described local virtual interrupt pin, according to place the physical node described local virtual interrupt pin of preserving and the corresponding relation processing the target vCPU that described local virtual interrupt pin is interrupted, the upper dig of described local virtual interrupt pin is delivered to target vCPU.
In the device that the present embodiment provides, same overall virtual interrupt pin in the ACPI table of subjunctive BIOS module write virtual memory corresponds to the local virtual interrupt pin on same physical node, namely an overall virtual interrupt pin goes simulation by the local virtual interrupt pin on same physical node, thus, operating system is when arranging this overall virtual interrupt pin, do not need to send the synchronous packet arranged to different nodes, improve the performance of distributed virtualization system.
For the phenomenon of the non-local process of avoidance breakout, namely avoid the phenomenon that the interruption that local interrupt pin produces is processed by the vCPU on the physical node of far-end, the second distributed virtual machine monitor also has following functions:
Described second distributed virtual machine monitor 73, also for when determining that according to described instruction described target vCPU place physical node is local physical node, preserve the corresponding relation of the target vCPU that described local virtual interrupt pin and described local virtual interrupt pin are interrupted.Wherein, processing the target vCPU that described local virtual interrupt pin is interrupted is vCPU in the instruction of operating system.
Described second distributed virtual machine monitor 73, also for when determining according to described instruction that described target vCPU place physical node is not local physical node and described local physical node comprises more than one vCPU, target vCPU described in described instruction is revised as a vCPU on described local physical node, the corresponding relation of described target vCPU after preserving local virtual interrupt pin and revising.
One of ordinary skill in the art will appreciate that: all or part of step realizing said method embodiment can have been come by the hardware that programmed instruction is relevant, aforesaid program can be stored in a computer read/write memory medium, this program, when performing, performs the step comprising said method embodiment; And aforesaid storage medium comprises: ROM, RAM, magnetic disc or CD etc. various can be program code stored medium.
Last it is noted that above embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to previous embodiment to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein portion of techniques feature; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the scope of various embodiments of the present invention technical scheme.