CN110780822B - Management container cloud local storage system and implementation method - Google Patents

Management container cloud local storage system and implementation method Download PDF

Info

Publication number
CN110780822B
CN110780822B CN201911031011.5A CN201911031011A CN110780822B CN 110780822 B CN110780822 B CN 110780822B CN 201911031011 A CN201911031011 A CN 201911031011A CN 110780822 B CN110780822 B CN 110780822B
Authority
CN
China
Prior art keywords
local
storage
local storage
persistent
block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911031011.5A
Other languages
Chinese (zh)
Other versions
CN110780822A (en
Inventor
石光银
蔡卫卫
高传集
孙思清
张晖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Cloud Information Technology Co Ltd
Original Assignee
Inspur Cloud Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Cloud Information Technology Co Ltd filed Critical Inspur Cloud Information Technology Co Ltd
Priority to CN201911031011.5A priority Critical patent/CN110780822B/en
Publication of CN110780822A publication Critical patent/CN110780822A/en
Application granted granted Critical
Publication of CN110780822B publication Critical patent/CN110780822B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0658Controller construction arrangements

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention particularly relates to a management container cloud local storage system and an implementation method. The management container cloud local storage system comprises a local storage manager and a Hyperkube module, wherein the local storage manager is used for managing local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes. According to the management container cloud local storage system and the implementation method, a plurality of local files or block storage devices can be managed, local persistent volumes can be automatically generated, and the data validity of a container using the local storage volumes can be ensured, so that the effective management of the container cloud local storage resources is realized, and a container cloud user can conveniently and quickly realize own micro-services.

Description

Management container cloud local storage system and implementation method
Technical Field
The invention relates to the technical field of container cloud local storage, in particular to a management container cloud local storage system and an implementation method.
Background
Container technology is a popular PaaS technology for providing micro services, while Kubernetes is the most popular container orchestration management technology in the near future, and various internet vendors have successively introduced a container cloud platform based on Kubernetes.
Kubernetes is a brand new distributed architecture leading scheme based on container technology, provides a series of functions of deployment operation, resource scheduling, service discovery, dynamic expansion and contraction and the like for containerized application on the basis of Docker technology, and improves convenience and high availability of large-scale container cluster management.
Based on the above situation, the invention provides a management container cloud local storage system and an implementation method.
Disclosure of Invention
The invention provides a simple and efficient management container cloud local storage system and an implementation method thereof for overcoming the defects of the prior art.
The invention is realized by the following technical scheme:
a management container cloud local storage system, characterized by: the system comprises a local storage manager and a Hyperkube module, wherein the local storage manager is used for managing local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes;
the local storage manager is provided with a local storage controller, a local storage finder, a local storage object operation event monitor, a local storage cleaner, a local storage cleaning task controller, a local storage deletion process management table, an API tool and a volume cache;
the Hyperkube module is provided with a storage plug-in manager, a local storage plug-in, a local storage installer, a local storage uninstaller, a local block storage mapper, a local block storage demapper, a block equipment installer, a local memory and an installation tool class.
The local storage controller is used for starting a local storage finder, operating an event monitor by a local storage object, setting a timing task, and setting a local storage cleaner and a local storage cleaning task controller;
the local storage discoverer is responsible for regularly scanning the local storage mounting catalog and creating a local persistence storage object when a new catalog or block storage is discovered;
the local storage object operation event monitor is responsible for monitoring events of the persistent storage object PV, and calling the volume cache to cache the information of the persistent storage object PV of the node where the local storage manager is located;
the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table are used for cleaning the PV mounting catalog of the persistent storage object at regular time;
the API tool is used for calling an application interface server, and the volume management tool is used for carrying out mounting operation on the catalogue;
the storage plug-in manager is used for loading a local storage plug-in;
the local storage installer and the local storage uninstaller are respectively responsible for carrying out volume installation and volume uninstalling local catalogue on the container group;
the local block storage mapper, the local block storage demapper and the block equipment loader are used for respectively loading or unloading the local persistent storage object of the block storage type to the container group;
the mounting tool class is used for realizing specific roll mounting and roll unloading functions.
The timing task set by the local storage controller is processed once every 10 seconds; the local storage discoverer scans the local storage mounting catalog once every 10 seconds, and the local storage cleaner, the local storage cleaning task controller and the local storage deletion process management table clean the persistent storage object (PV) mounting catalog once every 10 seconds.
The implementation method of the management container cloud local storage system comprises the following steps:
the method comprises the steps that firstly, a user configures a local storage provider, and a volume mounting catalog is built or block equipment is built to be connected to the local storage mounting catalog in a soft mode;
the API tool obtains SC information from a local storage provider and creates a local persistent storage object;
thirdly, creating a container group with a persistent storage object for applying for the PVC template by a user;
step four, the storage class acquires SC information from the API tool, generates a persistent storage object application PVC, and binds a local persistent storage object;
fifthly, the API tool obtains the node from the dispatcher according to the local persistent storage object;
sixth, the API tool creates a container group at the specified node using the kubelet component.
The local persistent storage object storage class uses kubernetes.io/no-provider for supporting the local persistent storage object; the volume binding model uses waitforfirstConsumer; the storage class configuration content of the local persistent storage object is the same, and different services can be distinguished through names.
The volume model of the local persistent storage object supports file system storage types and block storage types.
When the volume model of the local persistent storage object is the file system storage type, the container group mounts the local persistent storage object without performing the volume mount action.
The persistence storage object applies for PVC and the volume model of the container group is defaulted to be a file system storage type; when the local persistent storage object uses a block storage type, the persistent storage object application PVC must also use a block storage type.
When the volume model of the local persistent storage object is of a block storage type, the container group mounts the local persistent storage object to mount the local storage path to/var/lib/kubelet/plug/kubeletes.
The beneficial effects of the invention are as follows: according to the management container cloud local storage system and the implementation method, a plurality of local files or block storage devices can be managed, local persistent volumes can be automatically generated, and the data validity of a container using the local storage volumes can be ensured, so that the effective management of the container cloud local storage resources is realized, and a container cloud user can conveniently and quickly realize own micro-services.
Drawings
FIG. 1 is a schematic diagram of a management container cloud local storage system according to the present invention.
Fig. 2 is a schematic diagram of an implementation method of the management container cloud local storage system of the present invention.
Detailed Description
In order to make the technical problems, technical schemes and beneficial effects to be solved more clear, the invention is described in detail below with reference to the embodiments. It should be noted that the specific embodiments described herein are for the purpose of illustrating the invention only and are not to be construed as limiting the invention.
The management container cloud Local storage system comprises a Local storage manager (Local-static-provider) and a Hyperkube module, wherein the Local storage manager is used for managing Local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes;
the local storage manager is provided with a local storage controller (Localcontroller), a local storage finder (discover), a local storage object operation event monitor (Populator), a local storage cleaner (Deleteter), a local storage cleaning task controller (Jobcontroller), a local storage deletion process management table (Proctable), an API tool (APIUtil) and a volume cache (volume cache);
the Hyperkube module is provided with a storage plug-in manager (plug in-ctl-manager), a local storage plug-in (localvolumeplug in), a local storage installer (localvolumemount), a local storage uninstaller (localvolumeUnmounter), a local block storage mapper (localvolumeMapper), a local block storage demapper (localvolumeUnminger), a block equipment installer (deviceMount), a local storage (localvolumeme) and a mounting tool class (mount).
The local storage controller is used for starting a local storage finder, operating an event monitor by a local storage object, setting a timing task, and setting a local storage cleaner and a local storage cleaning task controller;
the local storage discoverer is responsible for periodically scanning a local storage mount directory (mount dir) and creating a local persistent storage object (local PV) of a file system storage type (Filesystem) or a Block storage type (Block) when a new directory or Block storage is discovered;
the local storage object operation event monitor is responsible for monitoring events of a persistent storage object (PV), calling a Volume cache (Volume cache) to cache persistent storage object (PV) information of a node where a local-static-provider is located;
the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table are used for cleaning a persistent storage object (PV) mounting catalog at regular time;
the API tool is used for calling an application interface server (API server), and the volume management tool (Volumeutil) is used for carrying out mounting operation on the catalogue;
the storage plug-in manager is used for loading a local storage plug-in;
the local storage installer and the local storage uninstaller are respectively responsible for performing volume mount (mount) and volume uninstall (unmount) local catalogs on the container group (pod);
the local block storage mapper, the local block storage demapper and the block device installer mount or unload a local persistent storage object (local PV) of a block storage type (block) to a container group (pod) respectively;
the mount tool class is used to implement specific volume mount (mount) and volume unload (unmount) functions.
The timing task set by the local storage controller is processed once every 10 seconds; the local storage discoverer scans the local storage mounting catalog once every 10 seconds, and the local storage cleaner, the local storage cleaning task controller and the local storage deletion process management table clean the persistent storage object (PV) mounting catalog once every 10 seconds.
The implementation method of the management container cloud local storage system comprises the following steps:
the method comprises the steps that firstly, a user configures a local storage provider (local-provider), and a volume mounting catalog is built or block equipment is flexibly connected to the local storage mounting catalog;
the API tool obtains SC information from a local storage provider and creates a local persistent storage object;
thirdly, creating a container group with a persistent storage object for applying for the PVC template by a user;
step four, the storage class acquires SC information from the API tool, generates a persistent storage object application PVC, and binds a local persistent storage object;
fifth, the API tool obtains the node from a scheduler (scheduler) according to the local persistent storage object;
sixth, the API tool creates a container group at the specified node using the kubelet component.
Local persistent storage object storage classes (local PVs) use kubernetes.io/no-provider for supporting local persistent storage objects (local PVs); the volume binding model volumeBindingMode uses waitfirstConsumer; the storage class of the local persistent storage object (local PV) is configured to be the same, and different services can be distinguished by name.
The volume model (volumeMode) of the local persistent storage object (local PV) supports file system storage types (filesystems) and Block storage types (blocks).
When the volume model (volumeMode) of the local persistent storage object (local PV) is the file system storage type (FileSystem), the container group mount local persistent storage object (local PV) does not do a volume mount (mount) action.
The persistent storage object applies for a volume model (volumeMode) of PVC and container groups, by default, is a file system storage type (Filesystem); when the local persistent storage object (local PV) uses a Block storage type (Block), the persistent storage object application PVC must also use a Block storage type (Block).
When the volume model (volumeMode) of the local persistent storage object (local PV) is of the Block storage type (Block), the container group mounts the local persistent storage object (local PV) to mount the local storage path (local path) to/var/lib/kubelet/plug/kubeletes.
The above examples are only one of the specific embodiments of the present invention, and the ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the scope of the present invention.

Claims (8)

1. A management container cloud local storage system, characterized by: the system comprises a local storage manager and a Hyperkube module, wherein the local storage manager is used for managing local storage resources, and the Hyperkube module is used for realizing the function of Kubernetes;
the local storage manager is provided with a local storage controller, a local storage finder, a local storage object operation event monitor, a local storage cleaner, a local storage cleaning task controller, a local storage deletion process management table, an API tool and a volume cache;
the Hyperkube module is provided with a storage plug-in manager, a local storage plug-in, a local storage installer, a local storage uninstaller, a local block storage mapper, a local block storage demapper, a block equipment installer, a local memory and an installation tool class;
the local storage controller is used for starting a local storage finder, operating an event monitor by a local storage object, setting a timing task, and setting a local storage cleaner and a local storage cleaning task controller;
the local storage discoverer is responsible for regularly scanning the local storage mounting catalog and creating a local persistence storage object when a new catalog or block storage is discovered;
the local storage object operation event monitor is responsible for monitoring events of the persistent storage object PV, and calling the volume cache to cache the information of the persistent storage object PV of the node where the local storage manager is located;
the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table are used for cleaning the PV mounting catalog of the persistent storage object at regular time;
the API tool is used for calling an application interface server, and the volume management tool is used for carrying out mounting operation on the catalogue;
the storage plug-in manager is used for loading a local storage plug-in;
the local storage installer and the local storage uninstaller are respectively responsible for carrying out volume installation and volume uninstalling local catalogue on the container group;
the local block storage mapper, the local block storage demapper and the block equipment loader are used for respectively loading or unloading the local persistent storage object of the block storage type to the container group;
the mounting tool class is used for realizing specific roll mounting and roll unloading functions.
2. The management container cloud local storage system of claim 1, wherein: the timing task set by the local storage controller is processed once every 10 seconds; the local storage discoverer scans the local storage mounting catalogue once every 10 seconds, and the local storage cleaner, the local storage cleaning task controller and the local storage deleting process management table clean the persistent storage object PV mounting catalogue once every 10 seconds.
3. A method for implementing the management container cloud local storage system according to claim 1 or 2, wherein,
the method comprises the following steps:
the method comprises the steps that firstly, a user configures a local storage provider, and a volume mounting catalog is built or block equipment is built to be connected to the local storage mounting catalog in a soft mode;
the API tool obtains SC information from a local storage provider and creates a local persistent storage object;
thirdly, creating a container group with a persistent storage object for applying for the PVC template by a user;
step four, the storage class acquires SC information from the API tool, generates a persistent storage object application PVC, and binds a local persistent storage object;
fifthly, the API tool obtains the node from the dispatcher according to the local persistent storage object;
sixth, the API tool creates a container group at the specified node using the kubelet component.
4. A method for implementing a management container cloud local storage system according to claim 3, wherein: the local persistent storage object storage class uses kubernetes.io/no-provider for supporting the local persistent storage object; the volume binding model uses waitforfirstConsumer; and the storage class configuration content of the local persistent storage object is the same, and different services are distinguished by names.
5. The method for implementing the management container cloud local storage system according to claim 4, wherein: the volume model of the local persistent storage object supports file system storage types and block storage types.
6. The method for implementing the management container cloud local storage system according to claim 5, wherein: when the volume model of the local persistent storage object is the file system storage type, the container group mounts the local persistent storage object without performing the volume mount action.
7. The method for implementing the management container cloud local storage system according to claim 5, wherein: the persistence storage object applies for PVC and the volume model of the container group is defaulted to be a file system storage type; when the local persistent storage object uses a block storage type, the persistent storage object application PVC must also use a block storage type.
8. The method for implementing the management container cloud local storage system according to claim 7, wherein: when the volume model of the local persistent storage object is of a block storage type, the container group mounts the local persistent storage object to mount the local storage path to/var/lib/kubelet/plug/kubeletes.
CN201911031011.5A 2019-10-28 2019-10-28 Management container cloud local storage system and implementation method Active CN110780822B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911031011.5A CN110780822B (en) 2019-10-28 2019-10-28 Management container cloud local storage system and implementation method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911031011.5A CN110780822B (en) 2019-10-28 2019-10-28 Management container cloud local storage system and implementation method

Publications (2)

Publication Number Publication Date
CN110780822A CN110780822A (en) 2020-02-11
CN110780822B true CN110780822B (en) 2023-09-19

Family

ID=69386938

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911031011.5A Active CN110780822B (en) 2019-10-28 2019-10-28 Management container cloud local storage system and implementation method

Country Status (1)

Country Link
CN (1) CN110780822B (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111753326B (en) * 2020-05-22 2024-02-13 湖南麒麟信安科技股份有限公司 Container cloud platform cloud storage resource encryption method, system and medium
CN111694641A (en) * 2020-06-16 2020-09-22 中电科华云信息技术有限公司 Storage management method and system for container application
CN111813503A (en) * 2020-07-11 2020-10-23 优品街科技有限公司 Micro-service application open system based on container cloud
CN111913665B (en) * 2020-07-30 2023-11-24 北京星辰天合科技股份有限公司 Storage volume mounting method and device and electronic equipment
CN112230857B (en) * 2020-10-22 2023-06-23 深圳软通动力信息技术有限公司 Hybrid cloud system, hybrid cloud disk application method and data storage method
CN112328170B (en) * 2020-10-22 2022-05-24 杭州朗澈科技有限公司 Cloud hard disk capacity expansion method and device, computer equipment and storage medium
CN112241342A (en) * 2020-10-22 2021-01-19 浪潮云信息技术股份公司 Method and system for realizing container cloud node autonomy
CN112612529A (en) * 2021-01-07 2021-04-06 上海明略人工智能(集团)有限公司 Shutdown method and device for Jupyter notewood operating system based on k8s
CN112527211A (en) * 2021-02-18 2021-03-19 北京城建设计发展集团股份有限公司 Local disk-based automatic storage supply method and system
CN113391952B (en) * 2021-06-07 2022-03-11 北京同创永益科技发展有限公司 Automatic backup method based on LVM snapshot in cloud native environment
CN114064207A (en) * 2021-11-10 2022-02-18 南京信易达计算技术有限公司 User data container storage method in cloud storage system based on customized LINUX architecture
CN114691050B (en) * 2022-05-26 2022-09-06 深圳前海环融联易信息科技服务有限公司 Cloud native storage method, device, equipment and medium based on kubernets
CN114995962B (en) * 2022-08-08 2022-12-02 浩鲸云计算科技股份有限公司 Method for loading emergency patches by initializing container

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106446275A (en) * 2016-10-21 2017-02-22 国云科技股份有限公司 Method for achieving container supporting file system expansion
CN106911802A (en) * 2017-04-18 2017-06-30 北京华云网际科技有限公司 The dispositions method and device of the management platform of distributed block storage system
CN109842642A (en) * 2017-11-24 2019-06-04 中兴通讯股份有限公司 A kind of dispositions method and device, containerization group system of containerization group system
CN109951525A (en) * 2019-02-18 2019-06-28 山东浪潮云信息技术有限公司 A kind of cloud disk carry system and method based on container

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160162209A1 (en) * 2014-12-05 2016-06-09 Hybrid Logic Ltd Data storage controller

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106446275A (en) * 2016-10-21 2017-02-22 国云科技股份有限公司 Method for achieving container supporting file system expansion
CN106911802A (en) * 2017-04-18 2017-06-30 北京华云网际科技有限公司 The dispositions method and device of the management platform of distributed block storage system
CN109842642A (en) * 2017-11-24 2019-06-04 中兴通讯股份有限公司 A kind of dispositions method and device, containerization group system of containerization group system
CN109951525A (en) * 2019-02-18 2019-06-28 山东浪潮云信息技术有限公司 A kind of cloud disk carry system and method based on container

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
何思政.面向容器云平台的集群资源调度管理器的设计与实现.中国优秀硕士学位论文全文数据库 信息科技辑.2017,(第12期),全文. *

Also Published As

Publication number Publication date
CN110780822A (en) 2020-02-11

Similar Documents

Publication Publication Date Title
CN110780822B (en) Management container cloud local storage system and implementation method
US10735345B2 (en) Orchestrating computing resources between different computing environments
CN108984560B (en) File storage method and device
CN103064742B (en) A kind of automatic deployment system and method for hadoop cluster
US9176786B2 (en) Dynamic and automatic colocation and combining of service providers and service clients in a grid of resources for performing a data backup function
CN111026414B (en) HDP platform deployment method based on kubernetes
US20100287280A1 (en) System and method for cloud computing based on multiple providers
CN104158886A (en) Flow-type performing method for applications
CN102591679A (en) Quick application software arrangement method for cluster system
CN106533713B (en) Application deployment method and device
CN106713493B (en) System and method for constructing distributed file in computer cluster environment
WO2013138091A1 (en) Network resource deployment for cloud-based services
CN1731738A (en) Automated quick deployment method for large-scale computer cluster node
CN101778132A (en) Method, system and device for network installation based on load balance
CN106911741B (en) Method for balancing virtual network management file downloading load and network management server
CN112799588B (en) Data storage method for loading container cluster application data by using external storage
CN108255500B (en) Cobbler-based operating system automatic installation method compatible with virtualization architecture
WO2016000303A1 (en) Resource allocation method and system, and computer storage medium
CN101146107B (en) A method and device for data download
CN103279374A (en) Operating system distributing technology based on MAC addresses
CN114629958B (en) Resource allocation method, device, electronic equipment and storage medium
CN114911421B (en) Data storage method, system, device and storage medium based on CSI plug-in
CN111404978A (en) Data storage method and cloud storage system
CN116155978A (en) Multi-registry adaptation method, device, electronic equipment and storage medium
CN111382132A (en) Medical image data cloud storage system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park

Applicant after: Inspur cloud Information Technology Co.,Ltd.

Address before: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park

Applicant before: Tidal Cloud Information Technology Co.,Ltd.

GR01 Patent grant
GR01 Patent grant