CN111209080A - Transparent transmission method for graphic processor - Google Patents
Transparent transmission method for graphic processor Download PDFInfo
- Publication number
- CN111209080A CN111209080A CN201911394285.0A CN201911394285A CN111209080A CN 111209080 A CN111209080 A CN 111209080A CN 201911394285 A CN201911394285 A CN 201911394285A CN 111209080 A CN111209080 A CN 111209080A
- Authority
- CN
- China
- Prior art keywords
- gpu
- virtual machine
- equipment
- transparent transmission
- cloud platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45579—I/O management, e.g. providing access to device drivers or storage
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a transparent transmission method of a graphic processor. The method comprises the following steps: determining whether GPU transparent transmission is enabled or not when a physical machine is added to the cloud platform; collecting information of all GPU equipment of the physical machine at regular time; setting a GPU equipment management page in a virtual machine page; selecting corresponding GPU equipment for the virtual machine, and transmitting the selected GPU equipment to the virtual machine; and starting the virtual machine and installing the drive of the GPU equipment corresponding to the virtual machine. By the method and the device, the GPU equipment can be transmitted to the virtual machine for use on the KVM virtualization platform of the CloudStack cloud platform.
Description
Technical Field
The present application relates to the field of Graphics Processing Unit (GPU) transparent transmission technology, and in particular, to a method for graphics processing unit transparent transmission.
Background
The transparent transmission of the Graphics Processing Unit (GPU) refers to the transparent transmission of physical GPU equipment to a virtual machine for use, and is mainly used for scenes with high requirements on graphics and images, such as high-definition video coding and decoding, 3D games, physical simulation and the like.
The CloudStack is an open-source cloud computing platform with high availability and expansibility, and as a popular cloud platform open-source framework, a user can quickly and conveniently establish own cloud service on the existing framework. However, on the cloudstock cloud computing platform, the GPU transparent transmission in the prior art only supports the XenServer and the VMware virtualization platform, but does not support the KVM virtualization platform.
KVM has become more and more popular with users as an excellent open source virtualization platform. However, the prior art solutions for passing through the GPU device to the virtual machine for KVM cannot be applied to the cloudstock cloud platform. That is to say, on the cloudstock cloud platform in the prior art, the KVM cannot pass through the GPU device for the virtual machine to use, and thus cannot meet the urgent requirement of the user using the KVM virtualization platform for high graphics and images.
Disclosure of Invention
In view of the above, the present invention provides a method for transparently transmitting a graphics processor, so that a GPU device can be transparently transmitted to a virtual machine for use on a KVM virtualization platform of a cloudstock cloud platform.
The technical scheme of the invention is realized as follows:
a method of transparent transmission for a graphics processor, the method comprising:
determining whether GPU transparent transmission is enabled or not when a physical machine is added to the cloud platform;
collecting information of all GPU equipment of the physical machine at regular time;
setting a GPU equipment management page in a virtual machine page;
selecting corresponding GPU equipment for the virtual machine, and transmitting the selected GPU equipment to the virtual machine;
and starting the virtual machine and installing the drive of the GPU equipment corresponding to the virtual machine.
Preferably, the determining whether to enable the GPU transparent transmission when the physical machine is added by the cloud platform comprises:
adding a selection field for a user to select whether GPU unvarnished transmission is enabled or not on a host page;
adding parameters related to the selection fields in an application program interface of the adding host;
saving parameters related to the selected field to a database;
when the GPU transparent transmission is determined to be started, calling a script in the cloud platform proxy setting script, and starting the GPU transparent transmission;
the virtual function input-output driver of the GPU device is unloaded from the physical machine.
Preferably, the field of the option added to the host page is "i/o memory management unit enabled status".
Preferably, the parameter added in the application program interface of the add host is iommuflag.
Preferably, the invoking the script in the cloud platform proxy setting script and the enabling of the GPU transparent transmission includes:
and calling a script managegpu.sh in the cloud platform proxy setting script, and setting the kernel guide parameter to be started.
Preferably, the collecting information of all GPU devices of the physical machine at regular time includes:
adding a timing task;
when the timing task is started, collecting information of all GPU equipment of the physical machine;
and saving the collected information of the GPU equipment into a database.
Preferably, the selecting the corresponding GPU device for the virtual machine and transmitting the selected GPU device to the virtual machine includes:
adding an additional device application program interface;
associating the virtual machine with the selected GPU device;
adding configuration parameters to the virtual machine configuration file associated with the selected GPU device.
Preferably, the configuration parameters include:
device mode, device type, drive name, and device address.
Preferably, the method further comprises:
and when the virtual machine does not need to reuse the GPU equipment, unloading the GPU equipment from the virtual machine.
Preferably, the unloading the GPU device from the virtual machine includes:
adding and removing an additional device application program interface;
the association of the virtual machine with the selected GPU device is deleted.
As can be seen from the above, in the method for transparent transmission of a graphics processor in the present invention, because it is determined whether to enable GPU transparent transmission when a physical machine is added to a cloud platform, and information of all GPU devices of the physical machine is collected at regular time, a GPU device management page is further set in a virtual machine page, a corresponding GPU device is selected for a virtual machine, and the selected GPU device is transmitted to the virtual machine, a virtual machine is started, and a drive of the GPU device corresponding to the virtual machine is installed, the GPU device can be transmitted to the virtual machine for use on a KVM virtualization platform of the cloudstock cloud platform, thereby realizing GPU transparent transmission based on the cloudstock cloud platform + KVM virtualization platform, and satisfying a number of urgent needs of users using the cloudstock cloud platform + KVM virtualization platform for graphics and images.
Drawings
FIG. 1 is a flowchart of a method for transparent transmission of a graphics processor according to an embodiment of the present invention.
Detailed Description
In order to make the technical scheme and advantages of the invention more apparent, the invention is further described in detail with reference to the accompanying drawings and specific embodiments.
FIG. 1 is a flowchart of a method for transparent transmission of a graphics processor according to an embodiment of the present invention. As shown in fig. 1, the method for transparent transmission of a graphics processor in the embodiment of the present invention includes the following steps:
and step 11, determining whether GPU transparent transmission is enabled or not when the physical machine is added to the cloud platform.
In the technical scheme of the invention, when a new physical machine is added to the cloud platform, whether GPU pass-through needs to be started or not needs to be determined, that is, whether an Input/Output memory management Unit (IOMMU) is started or not is selected.
In addition, in the technical solution of the present invention, the step 11 can be implemented by using various implementation methods. The technical solution of the present invention will be described in detail below by taking one specific implementation manner as an example.
For example, in one embodiment of the present invention, the step 11 may include the following steps:
and step 111, adding a selection field for the user to select whether the GPU unvarnished transmission is enabled on the host page.
For example, the fields of the option added to the host page may be, as an example: "IOMMU enabled status".
At step 112, add the parameters associated with the selection field in the add host's application program interface (addHost API).
For example, in one particular embodiment of the invention, a parameter iommuflag may be added to the addHost API, as an example.
Step 113, saving the parameters related to the selection fields in a database.
For example, in one embodiment of the invention, the added parameter iommuflag may be saved to a database, by way of example.
Step 114, when it is determined that the GPU transparent transmission needs to be enabled, calling a script in a cloud platform proxy-setup-agent (cloudstack-setup-agent) and starting the IOMMU.
For example, in one specific embodiment of the present invention, when it is determined that GPU transparent transmission needs to be enabled, a script (e.g., script manager pu. sh) may be called in the cloudstack-setup-agent, the kernel boot parameter may be set to on (e.g., add intel _ IOMMU ═ on to the grub boot file), and the IOMMU is turned on by passing "intel _ IOMMU ═ on" to the kernel boot parameter, i.e., GPU transparent transmission is enabled.
Step 115, uninstall the Virtual Function input/output (VFIO) driver of the GPU device from the physical machine.
In this step, the GPU vfio driver is offloaded from the physical machine so that the GPU device can be provided for use by the virtual machine. The VFIO is a framework that can safely expose device I/O, interrupt, DMA, etc. to a user space (userpace), so that device drivers can be completed in the user space. User space is directly accessed by the device, and the virtual machine device allocation can obtain higher IO performance.
Through the steps 111-115, whether GPU transparent transmission is started or not can be determined when the physical machine is added to the cloud platform, and the GPU transparent transmission is started when the GPU transparent transmission is determined to be needed.
And step 12, collecting information of all GPU equipment of the physical machine at regular time.
In the technical scheme of the invention, a timing task can be set to collect the information of all GPU equipment of the physical machine at regular time.
In addition, in the technical solution of the present invention, the step 12 can be implemented by using various implementation methods. The technical solution of the present invention will be described in detail below by taking one specific implementation manner as an example.
For example, in one embodiment of the present invention, the step 12 may include the following steps:
step 121, adding a timing task.
And step 122, when the timing task is started, collecting information of all GPU equipment of the physical machine.
In the technical scheme of the invention, when the timing task is started, the information of all GPU equipment of the physical machine can be collected by calling a command.
Step 123, saving the collected information of the GPU device in a database.
Through the steps 121-123, the information of all GPU equipment of the physical machine can be collected at regular time.
And step 13, setting a GPU equipment management page in the virtual machine page, wherein the GPU equipment management page is used for loading or unloading GPU equipment for the virtual machine by a user.
In the technical scheme of the invention, one GPU equipment management page can be arranged in the virtual machine page, and after the GPU equipment management page is arranged, a user can load or unload GPU equipment for the virtual machine in the GPU equipment management page.
And 14, selecting corresponding GPU equipment for the virtual machine, and transmitting the selected GPU equipment to the virtual machine.
In the technical scheme of the invention, after a user loads GPU equipment for the virtual machine in a GPU equipment management page, corresponding GPU equipment can be selected for the virtual machine, and then the selected GPU equipment is transmitted to the virtual machine.
In addition, in the technical solution of the present invention, the step 14 can be implemented by using various implementation methods. The technical solution of the present invention will be described in detail below by taking one specific implementation manner as an example.
For example, in one embodiment of the present invention, the step 14 may include the following steps:
at step 141, add an additional device application program interface (attachDevice API).
Step 142, associating the virtual machine with the selected GPU device.
Step 143, add configuration parameters to the virtual machine configuration file associated with the selected GPU device.
For example, in a specific embodiment of the present invention, the following configuration parameters may be added to the virtual machine configuration file associated with the selected GPU device: device mode, device type, drive name, and device address.
For example, configuration parameters such as device mode, device type, drive name, and device address may be added, as an example, in the following manner:
devicesBuilder.append("<hostdev mode＝'subsystem'type＝'pci'managed＝'yes'>\n")；
devicesBuilder.append("<driver name＝'vfio'/>\n")；
devicesBuilder.append("<source>\n")；
devicesBuilder.append("<address domain＝'0x0000'bus＝'0x"+firstId+"'slot＝'0x"+secondaryId+"'function＝'0x0'/>\n")；
devicesBuilder.append("</source>\n")。
through the steps 141-143, the corresponding GPU device can be selected for the virtual machine, and the selected GPU device is passed through to the virtual machine.
And step 15, starting the virtual machine, and installing the drive of the GPU equipment corresponding to the virtual machine.
In this step, the virtual machine may be started, and the driver of the GPU device corresponding to the virtual machine may be installed, so that the virtual machine may use the corresponding GPU device.
Therefore, through the steps 11 to 15, the GPU device can be passed through to the virtual machine for use on the KVM virtualization platform of the cloudstock cloud platform.
In addition, in the technical solution of the present invention, after the step 15, the method may further include:
and step 16, when the virtual machine does not need to reuse the GPU equipment, unloading the GPU equipment from the virtual machine.
In addition, in the technical solution of the present invention, the step 16 can be implemented by using various implementation methods. The technical solution of the present invention will be described in detail below by taking one specific implementation manner as an example.
For example, in a specific embodiment of the present invention, the unloading the GPU device from the virtual machine may include the following steps:
step 161, add detach add-on device application program interface (dettachDevice API).
Step 162, delete the association of the virtual machine with the selected GPU device.
In summary, in the technical solution of the present invention, when a physical machine is added to a cloud platform, it is determined whether GPU transparent transmission is enabled, and information of all GPU devices of the physical machine is collected at regular time, a GPU device management page is further set in the virtual machine page, a corresponding GPU device is selected for the virtual machine, the selected GPU device is transmitted to the virtual machine, the virtual machine is started, and a driver of the GPU device corresponding to the virtual machine is installed, so that the GPU device can be transmitted to the virtual machine for use on a KVM virtualization platform of the cloudstock cloud platform, thereby implementing GPU transparent transmission based on the cloudstock cloud platform + the KVM virtualization platform, and satisfying a number of urgent requirements of users using the cloudstock cloud platform + the KVM virtualization platform for graphics and images.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for transparent transmission of a graphics processor, the method comprising:
determining whether GPU transparent transmission is enabled or not when a physical machine is added to the cloud platform;
collecting information of all GPU equipment of the physical machine at regular time;
setting a GPU equipment management page in a virtual machine page;
selecting corresponding GPU equipment for the virtual machine, and transmitting the selected GPU equipment to the virtual machine;
and starting the virtual machine and installing the drive of the GPU equipment corresponding to the virtual machine.
2. The method of claim 1, wherein determining whether GPU pass-through is enabled when a physical machine is added by the cloud platform comprises:
adding a selection field for a user to select whether GPU unvarnished transmission is enabled or not on a host page;
adding parameters related to the selection fields in an application program interface of the adding host;
saving parameters related to the selected field to a database;
when the GPU transparent transmission is determined to be started, calling a script in the cloud platform proxy setting script, and starting the GPU transparent transmission;
the virtual function input-output driver of the GPU device is unloaded from the physical machine.
3. The method of claim 2, wherein:
the field of the option field added to the host page is "i/o memory management unit enable status".
4. The method of claim 2, wherein:
the added parameter in the application program interface of the adding host is iommuflag.
5. The method of claim 2, wherein invoking the script in the cloud platform proxy setup script, and enabling the GPU pass-through comprises:
and calling a script managegpu.sh in the cloud platform proxy setting script, and setting the kernel guide parameter to be started.
6. The method of claim 1, wherein the periodically collecting information for all GPU devices of a physical machine comprises:
adding a timing task;
when the timing task is started, collecting information of all GPU equipment of the physical machine;
and saving the collected information of the GPU equipment into a database.
7. The method of claim 1, wherein selecting the corresponding GPU device for the virtual machine and passing through the selected GPU device to the virtual machine comprises:
adding an additional device application program interface;
associating the virtual machine with the selected GPU device;
adding configuration parameters to the virtual machine configuration file associated with the selected GPU device.
8. The method of claim 7, wherein the configuration parameters comprise:
device mode, device type, drive name, and device address.
9. The method of claim 1, further comprising:
and when the virtual machine does not need to reuse the GPU equipment, unloading the GPU equipment from the virtual machine.
10. The method of claim 9, wherein offloading the GPU device from the virtual machine comprises:
adding and removing an additional device application program interface;
the association of the virtual machine with the selected GPU device is deleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911394285.0A CN111209080A (en) | 2019-12-30 | 2019-12-30 | Transparent transmission method for graphic processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911394285.0A CN111209080A (en) | 2019-12-30 | 2019-12-30 | Transparent transmission method for graphic processor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111209080A true CN111209080A (en) | 2020-05-29 |
Family
ID=70786536
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911394285.0A Pending CN111209080A (en) | 2019-12-30 | 2019-12-30 | Transparent transmission method for graphic processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111209080A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857961A (en) * | 2020-07-28 | 2020-10-30 | 山东超越数控电子股份有限公司 | Display card transparent transmission method, virtual desktop management system and computer equipment |
| CN112596859A (en) * | 2020-12-25 | 2021-04-02 | 创新科技术有限公司 | USB transparent transmission method and device |
| CN112667347A (en) * | 2020-09-24 | 2021-04-16 | 紫光云技术有限公司 | Method for transmitting GPU (graphics processing Unit) to cloud host for use in public cloud scene |
| CN113254158A (en) * | 2021-06-11 | 2021-08-13 | 苏州浪潮智能科技有限公司 | Deployment method and device of deep learning system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491188A (en) * | 2013-09-30 | 2014-01-01 | 上海沃帆信息科技有限公司 | Implementation method for enabling multiple users to share graphic workstation by using virtual desktop and GPU pass-through |
| CN106909443A (en) * | 2017-02-28 | 2017-06-30 | 郑州云海信息技术有限公司 | The collocation method and device of a kind of virtual machine |
| CN110308997A (en) * | 2019-07-11 | 2019-10-08 | 厦门厦科云海信息技术有限公司 | Method based on OpenStack platform management GPU equipment |
-
2019
- 2019-12-30 CN CN201911394285.0A patent/CN111209080A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491188A (en) * | 2013-09-30 | 2014-01-01 | 上海沃帆信息科技有限公司 | Implementation method for enabling multiple users to share graphic workstation by using virtual desktop and GPU pass-through |
| CN106909443A (en) * | 2017-02-28 | 2017-06-30 | 郑州云海信息技术有限公司 | The collocation method and device of a kind of virtual machine |
| CN110308997A (en) * | 2019-07-11 | 2019-10-08 | 厦门厦科云海信息技术有限公司 | Method based on OpenStack platform management GPU equipment |
Non-Patent Citations (5)
| Title |
|---|
| 293F65F8AF: "GPU及USB设备透传使用教程-ZStack", pages 14 - 19, Retrieved from the Internet <URL:《https://www.docin.com/p-2213950296.html》> * |
| ZSTACK云计算: "基于ZStack构建深度学习云平台", pages 5, Retrieved from the Internet <URL:《 https://zhuanlan.zhihu.com/p/57612014》> * |
| 关昆仑: "在云计算平台Openstack中利用GPU透传实现高性能计算", vol. 24, no. 6, pages 11 - 13 * |
| 勤菜鸟: "虚拟机显卡直通" * |
| 无: "ZStack Cloud 4.7.11>实践教程>私有云>GPU设备透传使用教程>典型使用流程", pages 1 - 8, Retrieved from the Internet <URL:《https://www.zstack.io/help/tutorials/vgpu_passthrough_tutorial/v4/4.html》> * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857961A (en) * | 2020-07-28 | 2020-10-30 | 山东超越数控电子股份有限公司 | Display card transparent transmission method, virtual desktop management system and computer equipment |
| CN112667347A (en) * | 2020-09-24 | 2021-04-16 | 紫光云技术有限公司 | Method for transmitting GPU (graphics processing Unit) to cloud host for use in public cloud scene |
| CN112596859A (en) * | 2020-12-25 | 2021-04-02 | 创新科技术有限公司 | USB transparent transmission method and device |
| CN113254158A (en) * | 2021-06-11 | 2021-08-13 | 苏州浪潮智能科技有限公司 | Deployment method and device of deep learning system |
| CN113254158B (en) * | 2021-06-11 | 2021-10-12 | 苏州浪潮智能科技有限公司 | Deployment method and device of deep learning system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107209683B (en) | Backup image restore | |
| CN111209080A (en) | Transparent transmission method for graphic processor | |
| US20210406388A1 (en) | Accessing file systems in a virtual environment | |
| US8151265B2 (en) | Apparatus for and method for real-time optimization of virtual machine input/output performance | |
| US9600369B2 (en) | Operating system recovery method and apparatus, and terminal device | |
| US20130055254A1 (en) | Methods and apparatuses for providing a virtual machine with dynamic assignment of a physical hardware resource | |
| KR102321913B1 (en) | Non-volatile memory device, and memory system having the same | |
| US9448807B2 (en) | Automatic creation, deployment, and upgrade of disk images | |
| US8813071B2 (en) | Storage reclamation systems and methods | |
| US20150160963A1 (en) | Scheduling of processes using a virtual file system | |
| WO2023179388A1 (en) | Hot migration method for virtual machine instance | |
| CN115988218B (en) | Virtualized video encoding and decoding system, electronic equipment and storage medium | |
| EP2905700A1 (en) | Operating system recovery method, apparatus and terminal device | |
| CN108292233B (en) | Application processor for starting virtual machine | |
| US20190377612A1 (en) | VCPU Thread Scheduling Method and Apparatus | |
| CN112352221A (en) | Shared memory mechanism to support fast transfer of SQ/CQ pair communications between SSD device drivers and physical SSDs in virtualized environments | |
| EP2466459A1 (en) | Seamless application integration apparatus and method | |
| US10185573B2 (en) | Caching based operating system installation | |
| US10467078B2 (en) | Crash dump extraction of guest failure | |
| CN110209354B (en) | Method, apparatus, device and medium for processing data | |
| CN115904761A (en) | System on chip, vehicle and video processing unit virtualization method | |
| US11263082B2 (en) | Data recovery of guest virtual machines | |
| US10565355B2 (en) | Techniques of managing licenses of virtual desktops | |
| CN115794132A (en) | Operating system deployment method, device, equipment and storage medium | |
| CN106886373B (en) | Physical machine and magnetic disk operation method and device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |