CN110780822B - Management container cloud local storage system and implementation method - Google Patents
Management container cloud local storage system and implementation method Download PDFInfo
- Publication number
- CN110780822B CN110780822B CN201911031011.5A CN201911031011A CN110780822B CN 110780822 B CN110780822 B CN 110780822B CN 201911031011 A CN201911031011 A CN 201911031011A CN 110780822 B CN110780822 B CN 110780822B
- Authority
- CN
- China
- Prior art keywords
- local
- storage
- local storage
- persistent
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0658—Controller construction arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention particularly relates to a management container cloud local storage system and an implementation method. The management container cloud local storage system comprises a local storage manager and a Hyperkube module, wherein the local storage manager is used for managing local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes. According to the management container cloud local storage system and the implementation method, a plurality of local files or block storage devices can be managed, local persistent volumes can be automatically generated, and the data validity of a container using the local storage volumes can be ensured, so that the effective management of the container cloud local storage resources is realized, and a container cloud user can conveniently and quickly realize own micro-services.
Description
Technical Field
The invention relates to the technical field of container cloud local storage, in particular to a management container cloud local storage system and an implementation method.
Background
Container technology is a popular PaaS technology for providing micro services, while Kubernetes is the most popular container orchestration management technology in the near future, and various internet vendors have successively introduced a container cloud platform based on Kubernetes.
Kubernetes is a brand new distributed architecture leading scheme based on container technology, provides a series of functions of deployment operation, resource scheduling, service discovery, dynamic expansion and contraction and the like for containerized application on the basis of Docker technology, and improves convenience and high availability of large-scale container cluster management.
Based on the above situation, the invention provides a management container cloud local storage system and an implementation method.
Disclosure of Invention
The invention provides a simple and efficient management container cloud local storage system and an implementation method thereof for overcoming the defects of the prior art.
The invention is realized by the following technical scheme:
a management container cloud local storage system, characterized by: the system comprises a local storage manager and a Hyperkube module, wherein the local storage manager is used for managing local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes;
the local storage manager is provided with a local storage controller, a local storage finder, a local storage object operation event monitor, a local storage cleaner, a local storage cleaning task controller, a local storage deletion process management table, an API tool and a volume cache;
the Hyperkube module is provided with a storage plug-in manager, a local storage plug-in, a local storage installer, a local storage uninstaller, a local block storage mapper, a local block storage demapper, a block equipment installer, a local memory and an installation tool class.
The local storage controller is used for starting a local storage finder, operating an event monitor by a local storage object, setting a timing task, and setting a local storage cleaner and a local storage cleaning task controller;
the local storage discoverer is responsible for regularly scanning the local storage mounting catalog and creating a local persistence storage object when a new catalog or block storage is discovered;
the local storage object operation event monitor is responsible for monitoring events of the persistent storage object PV, and calling the volume cache to cache the information of the persistent storage object PV of the node where the local storage manager is located;
the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table are used for cleaning the PV mounting catalog of the persistent storage object at regular time;
the API tool is used for calling an application interface server, and the volume management tool is used for carrying out mounting operation on the catalogue;
the storage plug-in manager is used for loading a local storage plug-in;
the local storage installer and the local storage uninstaller are respectively responsible for carrying out volume installation and volume uninstalling local catalogue on the container group;
the local block storage mapper, the local block storage demapper and the block equipment loader are used for respectively loading or unloading the local persistent storage object of the block storage type to the container group;
the mounting tool class is used for realizing specific roll mounting and roll unloading functions.
The timing task set by the local storage controller is processed once every 10 seconds; the local storage discoverer scans the local storage mounting catalog once every 10 seconds, and the local storage cleaner, the local storage cleaning task controller and the local storage deletion process management table clean the persistent storage object (PV) mounting catalog once every 10 seconds.
The implementation method of the management container cloud local storage system comprises the following steps:
the method comprises the steps that firstly, a user configures a local storage provider, and a volume mounting catalog is built or block equipment is built to be connected to the local storage mounting catalog in a soft mode;
the API tool obtains SC information from a local storage provider and creates a local persistent storage object;
thirdly, creating a container group with a persistent storage object for applying for the PVC template by a user;
step four, the storage class acquires SC information from the API tool, generates a persistent storage object application PVC, and binds a local persistent storage object;
fifthly, the API tool obtains the node from the dispatcher according to the local persistent storage object;
sixth, the API tool creates a container group at the specified node using the kubelet component.
The local persistent storage object storage class uses kubernetes.io/no-provider for supporting the local persistent storage object; the volume binding model uses waitforfirstConsumer; the storage class configuration content of the local persistent storage object is the same, and different services can be distinguished through names.
The volume model of the local persistent storage object supports file system storage types and block storage types.
When the volume model of the local persistent storage object is the file system storage type, the container group mounts the local persistent storage object without performing the volume mount action.
The persistence storage object applies for PVC and the volume model of the container group is defaulted to be a file system storage type; when the local persistent storage object uses a block storage type, the persistent storage object application PVC must also use a block storage type.
When the volume model of the local persistent storage object is of a block storage type, the container group mounts the local persistent storage object to mount the local storage path to/var/lib/kubelet/plug/kubeletes.
The beneficial effects of the invention are as follows: according to the management container cloud local storage system and the implementation method, a plurality of local files or block storage devices can be managed, local persistent volumes can be automatically generated, and the data validity of a container using the local storage volumes can be ensured, so that the effective management of the container cloud local storage resources is realized, and a container cloud user can conveniently and quickly realize own micro-services.
Drawings
FIG. 1 is a schematic diagram of a management container cloud local storage system according to the present invention.
Fig. 2 is a schematic diagram of an implementation method of the management container cloud local storage system of the present invention.
Detailed Description
In order to make the technical problems, technical schemes and beneficial effects to be solved more clear, the invention is described in detail below with reference to the embodiments. It should be noted that the specific embodiments described herein are for the purpose of illustrating the invention only and are not to be construed as limiting the invention.
The management container cloud Local storage system comprises a Local storage manager (Local-static-provider) and a Hyperkube module, wherein the Local storage manager is used for managing Local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes;
the local storage manager is provided with a local storage controller (Localcontroller), a local storage finder (discover), a local storage object operation event monitor (Populator), a local storage cleaner (Deleteter), a local storage cleaning task controller (Jobcontroller), a local storage deletion process management table (Proctable), an API tool (APIUtil) and a volume cache (volume cache);
the Hyperkube module is provided with a storage plug-in manager (plug in-ctl-manager), a local storage plug-in (localvolumeplug in), a local storage installer (localvolumemount), a local storage uninstaller (localvolumeUnmounter), a local block storage mapper (localvolumeMapper), a local block storage demapper (localvolumeUnminger), a block equipment installer (deviceMount), a local storage (localvolumeme) and a mounting tool class (mount).
The local storage controller is used for starting a local storage finder, operating an event monitor by a local storage object, setting a timing task, and setting a local storage cleaner and a local storage cleaning task controller;
the local storage discoverer is responsible for periodically scanning a local storage mount directory (mount dir) and creating a local persistent storage object (local PV) of a file system storage type (Filesystem) or a Block storage type (Block) when a new directory or Block storage is discovered;
the local storage object operation event monitor is responsible for monitoring events of a persistent storage object (PV), calling a Volume cache (Volume cache) to cache persistent storage object (PV) information of a node where a local-static-provider is located;
the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table are used for cleaning a persistent storage object (PV) mounting catalog at regular time;
the API tool is used for calling an application interface server (API server), and the volume management tool (Volumeutil) is used for carrying out mounting operation on the catalogue;
the storage plug-in manager is used for loading a local storage plug-in;
the local storage installer and the local storage uninstaller are respectively responsible for performing volume mount (mount) and volume uninstall (unmount) local catalogs on the container group (pod);
the local block storage mapper, the local block storage demapper and the block device installer mount or unload a local persistent storage object (local PV) of a block storage type (block) to a container group (pod) respectively;
the mount tool class is used to implement specific volume mount (mount) and volume unload (unmount) functions.
The timing task set by the local storage controller is processed once every 10 seconds; the local storage discoverer scans the local storage mounting catalog once every 10 seconds, and the local storage cleaner, the local storage cleaning task controller and the local storage deletion process management table clean the persistent storage object (PV) mounting catalog once every 10 seconds.
The implementation method of the management container cloud local storage system comprises the following steps:
the method comprises the steps that firstly, a user configures a local storage provider (local-provider), and a volume mounting catalog is built or block equipment is flexibly connected to the local storage mounting catalog;
the API tool obtains SC information from a local storage provider and creates a local persistent storage object;
thirdly, creating a container group with a persistent storage object for applying for the PVC template by a user;
step four, the storage class acquires SC information from the API tool, generates a persistent storage object application PVC, and binds a local persistent storage object;
fifth, the API tool obtains the node from a scheduler (scheduler) according to the local persistent storage object;
sixth, the API tool creates a container group at the specified node using the kubelet component.
Local persistent storage object storage classes (local PVs) use kubernetes.io/no-provider for supporting local persistent storage objects (local PVs); the volume binding model volumeBindingMode uses waitfirstConsumer; the storage class of the local persistent storage object (local PV) is configured to be the same, and different services can be distinguished by name.
The volume model (volumeMode) of the local persistent storage object (local PV) supports file system storage types (filesystems) and Block storage types (blocks).
When the volume model (volumeMode) of the local persistent storage object (local PV) is the file system storage type (FileSystem), the container group mount local persistent storage object (local PV) does not do a volume mount (mount) action.
The persistent storage object applies for a volume model (volumeMode) of PVC and container groups, by default, is a file system storage type (Filesystem); when the local persistent storage object (local PV) uses a Block storage type (Block), the persistent storage object application PVC must also use a Block storage type (Block).
When the volume model (volumeMode) of the local persistent storage object (local PV) is of the Block storage type (Block), the container group mounts the local persistent storage object (local PV) to mount the local storage path (local path) to/var/lib/kubelet/plug/kubeletes.
The above examples are only one of the specific embodiments of the present invention, and the ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the scope of the present invention.
Claims (8)
1. A management container cloud local storage system, characterized by: the system comprises a local storage manager and a Hyperkube module, wherein the local storage manager is used for managing local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes;
the local storage manager is provided with a local storage controller, a local storage finder, a local storage object operation event monitor, a local storage cleaner, a local storage cleaning task controller, a local storage deletion process management table, an API tool and a volume cache;
the Hyperkube module is provided with a storage plug-in manager, a local storage plug-in, a local storage installer, a local storage uninstaller, a local block storage mapper, a local block storage demapper, a block equipment installer, a local memory and an installation tool class;
the local storage controller is used for starting a local storage finder, operating an event monitor by a local storage object, setting a timing task, and setting a local storage cleaner and a local storage cleaning task controller;
the local storage discoverer is responsible for regularly scanning the local storage mounting catalog and creating a local persistence storage object when a new catalog or block storage is discovered;
the local storage object operation event monitor is responsible for monitoring events of the persistent storage object PV, and calling the volume cache to cache the information of the persistent storage object PV of the node where the local storage manager is located;
the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table are used for cleaning the PV mounting catalog of the persistent storage object at regular time;
the API tool is used for calling an application interface server, and the volume management tool is used for carrying out mounting operation on the catalogue;
the storage plug-in manager is used for loading a local storage plug-in;
the local storage installer and the local storage uninstaller are respectively responsible for carrying out volume installation and volume uninstalling local catalogue on the container group;
the local block storage mapper, the local block storage demapper and the block equipment loader are used for respectively loading or unloading the local persistent storage object of the block storage type to the container group;
the mounting tool class is used for realizing specific roll mounting and roll unloading functions.
2. The management container cloud local storage system of claim 1, wherein: the timing task set by the local storage controller is processed once every 10 seconds; the local storage discoverer scans the local storage mounting catalogue once every 10 seconds, and the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table clean the persistent storage object PV mounting catalogue once every 10 seconds.
3. A method for implementing the management container cloud local storage system according to claim 1 or 2, wherein,
the method comprises the following steps:
the method comprises the steps that firstly, a user configures a local storage provider, and a volume mounting catalog is built or block equipment is built to be connected to the local storage mounting catalog in a soft mode;
the API tool obtains SC information from a local storage provider and creates a local persistent storage object;
thirdly, creating a container group with a persistent storage object for applying for the PVC template by a user;
step four, the storage class acquires SC information from the API tool, generates a persistent storage object application PVC, and binds a local persistent storage object;
fifthly, the API tool obtains the node from the dispatcher according to the local persistent storage object;
sixth, the API tool creates a container group at the specified node using the kubelet component.
4. A method for implementing a management container cloud local storage system according to claim 3, wherein: the local persistent storage object storage class uses kubernetes.io/no-provider for supporting the local persistent storage object; the volume binding model uses waitforfirstConsumer; and the storage class configuration content of the local persistent storage object is the same, and different services are distinguished by names.
5. The method for implementing the management container cloud local storage system according to claim 4, wherein: the volume model of the local persistent storage object supports file system storage types and block storage types.
6. The method for implementing the management container cloud local storage system according to claim 5, wherein: when the volume model of the local persistent storage object is the file system storage type, the container group mounts the local persistent storage object without performing the volume mount action.
7. The method for implementing the management container cloud local storage system according to claim 5, wherein: the persistence storage object applies for PVC and the volume model of the container group is defaulted to be a file system storage type; when the local persistent storage object uses a block storage type, the persistent storage object application PVC must also use a block storage type.
8. The method for implementing the management container cloud local storage system according to claim 7, wherein: when the volume model of the local persistent storage object is of a block storage type, the container group mounts the local persistent storage object to mount the local storage path to/var/lib/kubelet/plug/kubeletes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911031011.5A CN110780822B (en) | 2019-10-28 | 2019-10-28 | Management container cloud local storage system and implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911031011.5A CN110780822B (en) | 2019-10-28 | 2019-10-28 | Management container cloud local storage system and implementation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110780822A CN110780822A (en) | 2020-02-11 |
| CN110780822B true CN110780822B (en) | 2023-09-19 |
Family
ID=69386938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911031011.5A Active CN110780822B (en) | 2019-10-28 | 2019-10-28 | Management container cloud local storage system and implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110780822B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111753326B (en) * | 2020-05-22 | 2024-02-13 | 湖南麒麟信安科技股份有限公司 | Container cloud platform cloud storage resource encryption method, system and medium |
| CN111694641A (en) * | 2020-06-16 | 2020-09-22 | 中电科华云信息技术有限公司 | Storage management method and system for container application |
| CN111813503A (en) * | 2020-07-11 | 2020-10-23 | 优品街科技有限公司 | Micro-service application open system based on container cloud |
| CN111913665B (en) * | 2020-07-30 | 2023-11-24 | 北京星辰天合科技股份有限公司 | Storage volume mounting method and device and electronic equipment |
| CN112230857B (en) * | 2020-10-22 | 2023-06-23 | 深圳软通动力信息技术有限公司 | Hybrid cloud system, hybrid cloud disk application method and data storage method |
| CN112328170B (en) * | 2020-10-22 | 2022-05-24 | 杭州朗澈科技有限公司 | Cloud hard disk capacity expansion method and device, computer equipment and storage medium |
| CN112241342A (en) * | 2020-10-22 | 2021-01-19 | 浪潮云信息技术股份公司 | Method and system for realizing container cloud node autonomy |
| CN112612529A (en) * | 2021-01-07 | 2021-04-06 | 上海明略人工智能(集团)有限公司 | Shutdown method and device for Jupyter notewood operating system based on k8s |
| CN112527211A (en) * | 2021-02-18 | 2021-03-19 | 北京城建设计发展集团股份有限公司 | Local disk-based automatic storage supply method and system |
| CN113391952B (en) * | 2021-06-07 | 2022-03-11 | 北京同创永益科技发展有限公司 | Automatic backup method based on LVM snapshot in cloud native environment |
| CN114064207A (en) * | 2021-11-10 | 2022-02-18 | 南京信易达计算技术有限公司 | User data container storage method in cloud storage system based on customized LINUX architecture |
| CN114691050B (en) * | 2022-05-26 | 2022-09-06 | 深圳前海环融联易信息科技服务有限公司 | Cloud native storage method, device, equipment and medium based on kubernets |
| CN114995962B (en) * | 2022-08-08 | 2022-12-02 | 浩鲸云计算科技股份有限公司 | Method for loading emergency patches by initializing container |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446275A (en) * | 2016-10-21 | 2017-02-22 | 国云科技股份有限公司 | Method for achieving container supporting file system expansion |
| CN106911802A (en) * | 2017-04-18 | 2017-06-30 | 北京华云网际科技有限公司 | The dispositions method and device of the management platform of distributed block storage system |
| CN109842642A (en) * | 2017-11-24 | 2019-06-04 | 中兴通讯股份有限公司 | A kind of dispositions method and device, containerization group system of containerization group system |
| CN109951525A (en) * | 2019-02-18 | 2019-06-28 | 山东浪潮云信息技术有限公司 | A kind of cloud disk carry system and method based on container |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160162209A1 (en) * | 2014-12-05 | 2016-06-09 | Hybrid Logic Ltd | Data storage controller |
-
2019
- 2019-10-28 CN CN201911031011.5A patent/CN110780822B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446275A (en) * | 2016-10-21 | 2017-02-22 | 国云科技股份有限公司 | Method for achieving container supporting file system expansion |
| CN106911802A (en) * | 2017-04-18 | 2017-06-30 | 北京华云网际科技有限公司 | The dispositions method and device of the management platform of distributed block storage system |
| CN109842642A (en) * | 2017-11-24 | 2019-06-04 | 中兴通讯股份有限公司 | A kind of dispositions method and device, containerization group system of containerization group system |
| CN109951525A (en) * | 2019-02-18 | 2019-06-28 | 山东浪潮云信息技术有限公司 | A kind of cloud disk carry system and method based on container |
Non-Patent Citations (1)
| Title |
|---|
| 何思政.面向容器云平台的集群资源调度管理器的设计与实现.中国优秀硕士学位论文全文数据库 信息科技辑.2017,(第12期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110780822A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110780822B (en) | Management container cloud local storage system and implementation method | |
| US10735345B2 (en) | Orchestrating computing resources between different computing environments | |
| CN108984560B (en) | File storage method and device | |
| CN103064742B (en) | A kind of automatic deployment system and method for hadoop cluster | |
| US9176786B2 (en) | Dynamic and automatic colocation and combining of service providers and service clients in a grid of resources for performing a data backup function | |
| CN111026414B (en) | HDP platform deployment method based on kubernetes | |
| US20100287280A1 (en) | System and method for cloud computing based on multiple providers | |
| CN104158886A (en) | Flow-type performing method for applications | |
| CN102591679A (en) | Quick application software arrangement method for cluster system | |
| CN106533713B (en) | Application deployment method and device | |
| CN106713493B (en) | System and method for constructing distributed file in computer cluster environment | |
| WO2013138091A1 (en) | Network resource deployment for cloud-based services | |
| CN1731738A (en) | Automated quick deployment method for large-scale computer cluster node | |
| CN101778132A (en) | Method, system and device for network installation based on load balance | |
| CN106911741B (en) | Method for balancing virtual network management file downloading load and network management server | |
| CN112799588B (en) | Data storage method for loading container cluster application data by using external storage | |
| CN108255500B (en) | Cobbler-based operating system automatic installation method compatible with virtualization architecture | |
| WO2016000303A1 (en) | Resource allocation method and system, and computer storage medium | |
| CN101146107B (en) | A method and device for data download | |
| CN103279374A (en) | Operating system distributing technology based on MAC addresses | |
| CN114629958B (en) | Resource allocation method, device, electronic equipment and storage medium | |
| CN114911421B (en) | Data storage method, system, device and storage medium based on CSI plug-in | |
| CN111404978A (en) | Data storage method and cloud storage system | |
| CN116155978A (en) | Multi-registry adaptation method, device, electronic equipment and storage medium | |
| CN111382132A (en) | Medical image data cloud storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant after: Inspur cloud Information Technology Co.,Ltd. Address before: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant before: Tidal Cloud Information Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |