CN117499396A - Data processing method and device based on cloud native platform system - Google Patents
Data processing method and device based on cloud native platform system Download PDFInfo
- Publication number
- CN117499396A CN117499396A CN202311586712.1A CN202311586712A CN117499396A CN 117499396 A CN117499396 A CN 117499396A CN 202311586712 A CN202311586712 A CN 202311586712A CN 117499396 A CN117499396 A CN 117499396A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- storage
- container
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
技术领域Technical field
本发明涉及计算机技术领域,尤其涉及一种基于云原生平台系统的数据处理的方法和装置。The present invention relates to the field of computer technology, and in particular, to a data processing method and device based on a cloud native platform system.
背景技术Background technique
随着云原生、分布式技术的普及,容器得到了广泛的应用,但由于容器属于“快速启动伸缩、快速销毁”的计算资源,容器不适合做数据的持久化存储,容器使用持久化卷用于数据的持久化存储。With the popularization of cloud-native and distributed technologies, containers have been widely used. However, because containers are computing resources with "quick startup, scaling, and quick destruction", containers are not suitable for persistent storage of data. Persistent volumes are used for containers. For persistent storage of data.
然而,容器使用的持久化卷,底层存储一种是使用容器所在计算节点的本地硬盘向该计算节点上的容器提供存储空间,然而本地硬盘没有副本保护机制,计算节点损坏时数据会丢失;另一种是使用额外的共享存储,通过NFS协议向容器提供存储空间,然而容器跨网络访问共享存储,会导致性能降低,且造成了本地硬盘资源浪费。However, the underlying storage of the persistent volume used by the container is to use the local hard disk of the computing node where the container is located to provide storage space for the container on the computing node. However, the local hard disk does not have a copy protection mechanism, and data will be lost when the computing node is damaged; in addition, One is to use additional shared storage to provide storage space to the container through the NFS protocol. However, when the container accesses the shared storage across the network, it will lead to performance degradation and a waste of local hard disk resources.
发明内容Contents of the invention
有鉴于此,本发明实施例提供一种基于云原生平台系统的数据处理的方法和装置,通过将容器集群与分布式存储系统混合部署,实现容器集群中数据的持久化存储,实现了数据的多副本保护,提高了数据可靠性,且充分利用容器集群的本地硬盘存储资源。In view of this, embodiments of the present invention provide a data processing method and device based on a cloud-native platform system. By hybrid deployment of a container cluster and a distributed storage system, persistent storage of data in the container cluster is achieved, and data storage is realized. Multi-copy protection improves data reliability and fully utilizes the local hard disk storage resources of the container cluster.
为实现上述目的,根据本发明实施例的一个方面,提供了一种云原生平台系统,包括:To achieve the above objectives, according to one aspect of an embodiment of the present invention, a cloud native platform system is provided, including:
容器集群和部署在所述容器集群上的分布式存储系统;Container clusters and distributed storage systems deployed on the container clusters;
所述容器集群包括容器主节点和容器从节点;The container cluster includes a container master node and a container slave node;
所述分布式存储系统包括存储主节点、元数据节点和数据子节点;所述存储主节点部署在所述容器主节点上;The distributed storage system includes a storage master node, a metadata node and a data sub-node; the storage master node is deployed on the container master node;
所述元数据节点和数据子节点部署在所述容器从节点上,所述元数据节点和所述数据子节点作为所述容器从节点的存储空间,形成存储资源池。The metadata node and the data sub-node are deployed on the container slave node. The metadata node and the data sub-node serve as the storage space of the container slave node to form a storage resource pool.
可选地,所述数据子节点存储了数据标识和与所述数据标识对应的存储数据,所述元数据节点包括数据标识和数据子节点的对应关系。Optionally, the data sub-node stores a data identifier and stored data corresponding to the data identifier, and the metadata node includes a corresponding relationship between the data identifier and the data sub-node.
可选地,所述分布式存储系统还包括存储服务协议,所述存储服务协议部署在所述容器从节点上,用于进行所述存储主节点与所述元数据节点和所述数据子节点的存储协议的转换。Optionally, the distributed storage system also includes a storage service protocol, which is deployed on the container slave node and is used to carry out communication between the storage master node, the metadata node and the data sub-node. Conversion of storage protocols.
可选地,所述元数据节点和所述数据子节点分离部署在所述容器从节点上,所述元数据节点的数量和所述数据子节点的数量根据所述容器从节点的数量确定。Optionally, the metadata node and the data sub-node are deployed separately on the container slave node, and the number of the metadata node and the data sub-node is determined according to the number of the container slave node.
根据本发明实施例的另一方面,提供了一种基于云原生平台系统的数据处理方法,包括:According to another aspect of the embodiment of the present invention, a data processing method based on a cloud native platform system is provided, including:
利用云原生平台系统的容器主节点接收客户端发送的数据存储请求,所述数据存储请求包括数据标识和与所述数据标识对应的待存储数据;Utilize the container master node of the cloud native platform system to receive a data storage request sent by the client, where the data storage request includes a data identifier and data to be stored corresponding to the data identifier;
利用所述存储主节点根据存储资源池确定与所述待存储数据对应的多个目标数据子节点;Using the storage master node to determine multiple target data sub-nodes corresponding to the data to be stored according to the storage resource pool;
利用所述存储资源池获取所述待存储数据的多个副本;Using the storage resource pool to obtain multiple copies of the data to be stored;
利用所述容器从节点将所述数据标识和所述待存储数据的副本存储在所述目标数据子节点上。The container slave node is used to store the data identifier and a copy of the data to be stored on the target data sub-node.
可选地,利用所述容器从节点将所述数据标识和所述待存储数据的副本存储在所述目标数据子节点上:Optionally, use the container slave node to store the data identifier and a copy of the data to be stored on the target data sub-node:
利用所述容器从节点将所述数据标识、所述待存储数据的副本和所述目标数据子节点的对应关系存储在元数据节点上。The container slave node is used to store the corresponding relationship between the data identifier, the copy of the data to be stored, and the target data sub-node on the metadata node.
可选地,所述方法还包括:Optionally, the method also includes:
利用所述容器主节点接收客户端发送的数据查询请求,所述数据查询请求指示了数据标识;Utilize the container master node to receive a data query request sent by the client, where the data query request indicates a data identifier;
利用所述存储主节点确定与所述数据查询请求对应的目标元数据节点;Using the storage master node to determine the target metadata node corresponding to the data query request;
利用所述存储资源池根据所述目标元数据节点确定与所述数据标识对应的多个目标数据子节点;Using the storage resource pool to determine multiple target data sub-nodes corresponding to the data identifier according to the target metadata node;
利用所述容器从节点从多个目标数据子节点中获取与所述数据查询请求对应的查询结果。The container slave node is used to obtain query results corresponding to the data query request from multiple target data sub-nodes.
根据本发明实施例的再一个方面,提供了一种基于云原生平台系统的数据处理的装置,包括:According to another aspect of the embodiment of the present invention, a data processing device based on a cloud native platform system is provided, including:
接收模块,利用云原生平台系统的容器主节点接收客户端发送的数据存储请求,所述数据存储请求包括数据标识和与所述数据标识对应的待存储数据;The receiving module uses the container master node of the cloud native platform system to receive the data storage request sent by the client, where the data storage request includes a data identifier and the data to be stored corresponding to the data identifier;
确定模块,利用所述存储主节点根据存储资源池确定与所述待存储数据对应的多个目标数据子节点;A determination module that uses the storage master node to determine multiple target data sub-nodes corresponding to the data to be stored according to the storage resource pool;
获取模块,利用所述存储资源池获取所述待存储数据的多个副本;An acquisition module that uses the storage resource pool to acquire multiple copies of the data to be stored;
存储模块,利用所述容器从节点将所述数据标识和所述待存储数据的副本存储在所述目标数据子节点上。A storage module that uses the container slave node to store the data identifier and a copy of the data to be stored on the target data sub-node.
根据本发明实施例的另一个方面,提供了一种电子设备,包括:According to another aspect of the embodiment of the present invention, an electronic device is provided, including:
一个或多个处理器;one or more processors;
存储装置,用于存储一个或多个程序,a storage device for storing one or more programs,
当所述一个或多个程序被所述一个或多个处理器执行,使得所述一个或多个处理器实现本发明提供的基于云原生平台系统的数据处理的方法。When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data processing method based on the cloud native platform system provided by the present invention.
根据本发明实施例的还一个方面,提供了一种计算机可读介质,其上存储有计算机程序,所述程序被处理器执行时实现本发明提供的基于云原生平台系统的数据处理的方法。According to yet another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method based on the cloud native platform system provided by the present invention is implemented.
上述发明中的一个实施例具有如下优点或有益效果:本发明实施例提供的云原生平台系统以及基于云原生平台系统的数据处理方法,该云原生平台系统通过将分布式存储系统的组件容器化,将容器集群与分布式存储系统混合部署,充分利用容器集群中本地硬盘的存储能力,克服了容器数据持久化依赖外部存储的问题,并实现了应用的数据存储和计算的分离,降低了存储成本和运维复杂度;分布式存储的存储资源池可以作为共享存储用于容器集群的持久化存储,形成融合的云原生容器平台;该数据处理的方法能够实现数据的多副本保护,提高数据可靠性,且能够有效避免跨网络数据访问造成的开销和延迟。One embodiment of the above invention has the following advantages or beneficial effects: the cloud native platform system and the data processing method based on the cloud native platform system provided by the embodiment of the present invention, the cloud native platform system containerizes the components of the distributed storage system , hybrid deployment of container clusters and distributed storage systems, making full use of the storage capabilities of local hard disks in the container cluster, overcoming the problem of container data persistence relying on external storage, and realizing the separation of application data storage and computing, reducing storage costs Cost and operation and maintenance complexity; the storage resource pool of distributed storage can be used as shared storage for persistent storage of container clusters to form a converged cloud-native container platform; this data processing method can achieve multiple copies of data protection and improve data reliability, and can effectively avoid the overhead and delay caused by cross-network data access.
上述的非惯用的可选方式所具有的进一步效果将在下文中结合具体实施方式加以说明。Further effects of the above-mentioned non-conventional optional methods will be described below in conjunction with specific implementations.
附图说明Description of drawings
附图用于更好地理解本发明,不构成对本发明的不当限定。其中:The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. in:
图1是根据本发明实施例的一种基于云原生平台系统的数据处理方法的主要流程的示意图;Figure 1 is a schematic diagram of the main flow of a data processing method based on a cloud native platform system according to an embodiment of the present invention;
图2是根据本发明实施例的另一种基于云原生平台系统的数据处理方法的主要流程的示意图;Figure 2 is a schematic diagram of the main flow of another data processing method based on a cloud native platform system according to an embodiment of the present invention;
图3是根据本发明实施例的一种云原生平台系统的架构示意图;Figure 3 is a schematic architectural diagram of a cloud native platform system according to an embodiment of the present invention;
图4是根据本发明实施例的基于云原生平台系统的数据处理的装置的主要模块的示意图;Figure 4 is a schematic diagram of the main modules of a data processing device based on a cloud native platform system according to an embodiment of the present invention;
图5是本发明实施例可以应用于其中的示例性系统架构图;Figure 5 is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
图6是适于用来实现本发明实施例的终端设备或服务器的计算机系统的结构示意图。FIG. 6 is a schematic structural diagram of a computer system suitable for implementing a terminal device or server according to an embodiment of the present invention.
具体实施方式Detailed ways
以下结合附图对本发明的示范性实施例做出说明,其中包括本发明实施例的各种细节以助于理解,应当将它们认为仅仅是示范性的。因此,本领域普通技术人员应当认识到,可以对这里描述的实施例做出各种改变和修改,而不会背离本发明的范围和精神。同样,为了清楚和简明,以下的描述中省略了对公知功能和结构的描述。Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.
根据本发明实施例的一个方面,提供了一种云原生平台系统,包括:容器集群和部署在容器集群上的分布式存储系统;容器集群包括容器主节点和容器从节点;分布式存储系统包括存储主节点、元数据节点和数据子节点;存储主节点部署在容器主节点上;元数据节点和数据子节点部署在容器从节点上,元数据节点和数据子节点作为容器从节点的存储空间,形成存储资源池。According to one aspect of an embodiment of the present invention, a cloud native platform system is provided, including: a container cluster and a distributed storage system deployed on the container cluster; the container cluster includes a container master node and a container slave node; the distributed storage system includes Storage master node, metadata node and data sub-node; storage master node is deployed on the container master node; metadata node and data sub-node are deployed on the container slave node, and the metadata node and data sub-node serve as the storage space of the container slave node , forming a storage resource pool.
在本发明实施例中,该云原生平台系统通过容器与分布式存储融合部署实现。该云原生平台系统包括容器集群和部署在容器集群上的分布式存储系统,在容器集群中,包括容器主节点和容器从节点,容器集群为k8s(Kubernetes,一种开源的容器编排平台)集群,容器主节点即k8s集群的master节点,用于容器集群的调度、管理和运维,容器从节点即k8s集群的worker节点。分布式存储系统即分布式存储集群包括存储主节点、元数据节点和数据子节点,存储主节点即资源管理节点用于调度、管理存储节点,即元数据节点和数据子节点,存储主节点可以部署在k8s集群的任意容器主节点上,即将k8s的master节点和分布式存储系统的存储主节点混合部署,分布式存储系统的存储主节点作为容器服务部署在容器主节点上。In this embodiment of the present invention, the cloud native platform system is implemented through integrated deployment of containers and distributed storage. The cloud native platform system includes a container cluster and a distributed storage system deployed on the container cluster. The container cluster includes a container master node and a container slave node. The container cluster is a k8s (Kubernetes, an open source container orchestration platform) cluster. , the container master node is the master node of the k8s cluster, which is used for the scheduling, management and operation and maintenance of the container cluster, and the container slave node is the worker node of the k8s cluster. The distributed storage system, that is, the distributed storage cluster, includes storage master nodes, metadata nodes and data sub-nodes. The storage master node, which is the resource management node, is used to schedule and manage storage nodes, that is, metadata nodes and data sub-nodes. The storage master node can Deployed on any container master node in the k8s cluster, that is, the k8s master node and the storage master node of the distributed storage system are mixed and deployed. The storage master node of the distributed storage system is deployed on the container master node as a container service.
在本发明实施例中,元数据节点和数据子节点部署在容器从节点上,元数据节点部署在一个或多个容器从节点上,数据子节点部署在各个容器从节点上。每个服务器节点包括系统盘、缓存盘和数据盘,元数据节点和数据子节点分散在容器集群的容器从节点即worker节点上,元数据节点和数据子节点分离部署,元数据节点和数据子节点作为容器从节点的存储空间,存储空间包括缓存盘和数据盘,即元数据节点和数据子节点接管容器从节点的缓存盘和数据盘,从而使得元数据节点和数据子节点形成跨节点的分布式存储资源池,对外暴露POSIX和Object API两个接口,以供k8s集群调用。通过将分布式存储系统的组件部署在k8s集群中,通过device plugin(一个grpc服务)的方式拦截对本地Data硬盘的访问,形成一个集群内的分布式存储资源池,将存储资源池挂载给k8s集群,从而解决了原有本地硬盘数据可靠性的问题,也能充分利用每个计算节点的本地硬盘空间。In this embodiment of the present invention, metadata nodes and data subnodes are deployed on container slave nodes, metadata nodes are deployed on one or more container slave nodes, and data subnodes are deployed on each container slave node. Each server node includes a system disk, cache disk and data disk. Metadata nodes and data sub-nodes are scattered on the container slave node of the container cluster, that is, the worker node. Metadata nodes and data sub-nodes are deployed separately. Metadata nodes and data sub-nodes are deployed separately. The node serves as the storage space of the container slave node. The storage space includes cache disk and data disk. That is, the metadata node and data sub-node take over the cache disk and data disk of the container slave node, so that the metadata node and data sub-node form a cross-node network. The distributed storage resource pool exposes two interfaces, POSIX and Object API, for calls by the k8s cluster. By deploying the components of the distributed storage system in the k8s cluster, intercepting access to the local Data hard disk through device plugin (a grpc service), a distributed storage resource pool in the cluster is formed, and the storage resource pool is mounted to The k8s cluster solves the original problem of local hard disk data reliability and can also make full use of the local hard disk space of each computing node.
在本发明实施例中,数据子节点存储了数据标识和与数据标识对应的存储数据,元数据节点包括数据标识和数据子节点的对应关系。通过元数据节点可以获取与数据标识对应的数据子节点,进而获取与数据标识对应的存储数据。In the embodiment of the present invention, the data sub-node stores the data identifier and the stored data corresponding to the data identifier, and the metadata node includes the corresponding relationship between the data identifier and the data sub-node. Through the metadata node, the data sub-node corresponding to the data identifier can be obtained, and then the stored data corresponding to the data identifier can be obtained.
在本发明实施例中,分布式存储系统还包括存储服务协议(存储Service),存储服务协议部署在容器从节点上,用于进行存储主节点与元数据节点和数据子节点的存储协议的转换。该存储协议用于存储协议的转换,从存储主节点中获取存储卷信息,以及直接与元数据节点和数据子节点进行通信。通过存储服务协议实现与容器集群的良好的兼容性。In the embodiment of the present invention, the distributed storage system also includes a storage service protocol (storage service). The storage service protocol is deployed on the container slave node and is used to convert the storage protocol between the storage master node and the metadata node and data sub-node. . This storage protocol is used for storage protocol conversion, obtaining storage volume information from the storage master node, and communicating directly with metadata nodes and data subnodes. Good compatibility with container clusters is achieved through the storage service protocol.
在本发明实施例中,元数据节点和数据子节点分离部署在容器从节点上,元数据节点的数量和数据子节点的数量根据容器从节点的数量确定。存储服务协议、元数据节点和数据子节点均具有高扩展性,将元数据节点和数据子节点进行分离部署,根据容器集群中容器从节点的数量、存储容量、访问量的增加,分别扩展元数据节点和数据子节点以及数据存储服务协议的个数,在多个计算节点即服务器节点上水平扩展,从而实现将应用的数据访问请求分散在不同的元数据节点上,规避数据访问热点,保证读写性能,避免性能瓶颈。In this embodiment of the present invention, metadata nodes and data sub-nodes are separately deployed on container slave nodes, and the number of metadata nodes and data sub-nodes is determined based on the number of container slave nodes. The storage service protocol, metadata nodes and data sub-nodes all have high scalability. Metadata nodes and data sub-nodes are deployed separately. According to the increase in the number of container slave nodes, storage capacity and access volume in the container cluster, the metadata nodes and data sub-nodes are expanded respectively. The number of data nodes, data sub-nodes, and data storage service protocols is horizontally expanded on multiple computing nodes, that is, server nodes, thereby dispersing application data access requests on different metadata nodes, avoiding data access hotspots, and ensuring Read and write performance to avoid performance bottlenecks.
图1是根据本发明实施例的一种基于云原生平台系统的数据处理方法的主要流程的示意图,如图1所示,该基于云原生平台系统的数据处理方法,包括以下步骤:Figure 1 is a schematic diagram of the main flow of a data processing method based on a cloud native platform system according to an embodiment of the present invention. As shown in Figure 1, the data processing method based on a cloud native platform system includes the following steps:
步骤S101:利用云原生平台系统的容器主节点接收客户端发送的数据存储请求,数据存储请求包括数据标识和与数据标识对应的待存储数据;Step S101: Use the container master node of the cloud native platform system to receive the data storage request sent by the client. The data storage request includes a data identifier and the data to be stored corresponding to the data identifier;
步骤S102:利用存储主节点根据存储资源池确定与待存储数据对应的多个目标数据子节点;Step S102: Use the storage master node to determine multiple target data sub-nodes corresponding to the data to be stored according to the storage resource pool;
步骤S103:利用存储资源池获取待存储数据的多个副本;Step S103: Use the storage resource pool to obtain multiple copies of the data to be stored;
步骤S104:利用容器从节点将数据标识和待存储数据的副本存储在目标数据子节点上。Step S104: Use the container slave node to store the data identifier and a copy of the data to be stored on the target data child node.
在本发明实施例中,该数据处理的方法基于云原生平台系统实现。通过容器主节点接收客户端发送的数据存储请求,客户端可以为应用的存储客户端,容器主节点通过存储服务协议,将数据存储请求发送至容器从节点,数据存储请求包括数据标识和对应的待存储数据,数据标识可以为应用标识。利用存储主节点根据存储资源池确定与待存储数据对应的多个目标数据子节点,然后通过存储资源池获取待存储数据的多个副本,从而利用容器从节点将待存储数据的副本存储在目标数据子节点上,通过存储资源池实现待存储数据的副本保护机制,能够有效避免待存储数据的丢失。In this embodiment of the present invention, the data processing method is implemented based on a cloud native platform system. The container master node receives the data storage request sent by the client. The client can be the storage client of the application. The container master node sends the data storage request to the container slave node through the storage service protocol. The data storage request includes the data identifier and the corresponding For the data to be stored, the data identifier can be an application identifier. Use the storage master node to determine multiple target data sub-nodes corresponding to the data to be stored based on the storage resource pool, and then obtain multiple copies of the data to be stored through the storage resource pool, thereby using the container slave node to store copies of the data to be stored in the target On the data sub-nodes, the storage resource pool implements a copy protection mechanism for the data to be stored, which can effectively avoid the loss of the data to be stored.
在本发明实施例中,利用容器从节点将数据标识和待存储数据的副本存储在目标数据子节点上之后,还包括:利用容器从节点将数据标识、待存储数据的副本和目标数据子节点的对应关系存储在元数据节点上。即,通过数据标识可以确定对应的目标数据子节点,进而可以从目标数据子节点获取待存储数据。In the embodiment of the present invention, after using the container slave node to store the data identifier and the copy of the data to be stored on the target data sub-node, it also includes: using the container slave node to store the data identifier, the copy of the data to be stored and the target data sub-node. The corresponding relationship is stored on the metadata node. That is, the corresponding target data sub-node can be determined through the data identification, and then the data to be stored can be obtained from the target data sub-node.
在本发明实施例中,如图2所示,该数据处理的方法还包括:In the embodiment of the present invention, as shown in Figure 2, the data processing method further includes:
步骤S201:利用容器主节点接收客户端发送的数据查询请求,数据查询请求指示了数据标识;Step S201: Use the container master node to receive the data query request sent by the client, where the data query request indicates the data identifier;
步骤S202:利用存储主节点确定与数据查询请求对应的目标元数据节点;Step S202: Use the storage master node to determine the target metadata node corresponding to the data query request;
步骤S203:利用存储资源池根据目标元数据节点确定与数据标识对应的多个目标数据子节点;Step S203: Use the storage resource pool to determine multiple target data sub-nodes corresponding to the data identifier according to the target metadata node;
步骤S204:利用容器从节点从多个目标数据子节点中获取与数据查询请求对应的查询结果。Step S204: Use the container slave node to obtain query results corresponding to the data query request from multiple target data sub-nodes.
在本发明实施例中,在云原生平台系统进行数据查询时,利用容器主节点接收客户端发送的数据查询请求,然后容器主节点将数据查询请求发送至容器从节点,以向容器从节点的元数据节点发送数据查询请求,该数据查询请求包括数据标识,利用存储主节点确定目标元数据节点,根据数据标识和目标元数据节点,可以确定出存储资源池中与数据标识对应的各个目标数据子节点,每个目标数据子节点中存储了与数据标识对应的数据副本,通过对应的容器从节点可以从对应的目标数据子节点中获取查询结果,并将查询结果返回至容器主节点,以使容器主节点将查询结果返回至客户端。In the embodiment of the present invention, when the cloud native platform system performs data query, the container master node is used to receive the data query request sent by the client, and then the container master node sends the data query request to the container slave node, so as to send the data query request to the container slave node. The metadata node sends a data query request, which includes a data identifier. The storage master node is used to determine the target metadata node. According to the data identifier and the target metadata node, each target data corresponding to the data identifier in the storage resource pool can be determined. Sub-nodes, each target data sub-node stores a data copy corresponding to the data identifier. The query results can be obtained from the corresponding target data sub-nodes through the corresponding container slave nodes, and the query results are returned to the container master node to Causes the container master node to return query results to the client.
本发明实施例提供的云原生平台系统以及基于云原生平台系统的数据处理方法,该云原生平台系统通过将分布式存储系统的组件容器化,将容器集群与分布式存储系统混合部署,充分利用容器集群中本地硬盘的存储能力,克服了容器数据持久化依赖外部存储的问题,并实现了应用的数据存储和计算的分离,降低了存储成本和运维复杂度;分布式存储的存储资源池可以作为共享存储用于容器集群的持久化存储,形成融合的云原生容器平台;该数据处理的方法能够实现数据的多副本保护,提高数据可靠性,且能够有效避免跨网络数据访问造成的开销和延迟。The cloud native platform system and the data processing method based on the cloud native platform system provided by the embodiments of the present invention. The cloud native platform system containerizes the components of the distributed storage system and hybridly deploys the container cluster and the distributed storage system to make full use of The storage capacity of the local hard disk in the container cluster overcomes the problem of container data persistence relying on external storage, and realizes the separation of application data storage and computing, reducing storage costs and operation and maintenance complexity; distributed storage storage resource pool It can be used as shared storage for persistent storage of container clusters to form a converged cloud-native container platform; this data processing method can achieve multi-copy protection of data, improve data reliability, and can effectively avoid the overhead caused by cross-network data access. and delay.
根据本发明实施例的再一个方面,如图4所示,提供了一种基于云原生平台系统的数据处理的装置400,包括:According to another aspect of the embodiment of the present invention, as shown in Figure 4, a data processing device 400 based on a cloud native platform system is provided, including:
接收模块401,利用云原生平台系统的容器主节点接收客户端发送的数据存储请求,数据存储请求包括数据标识和与数据标识对应的待存储数据;The receiving module 401 uses the container master node of the cloud native platform system to receive the data storage request sent by the client. The data storage request includes a data identifier and the data to be stored corresponding to the data identifier;
确定模块402,利用存储主节点根据存储资源池确定与待存储数据对应的多个目标数据子节点;The determination module 402 uses the storage master node to determine multiple target data sub-nodes corresponding to the data to be stored according to the storage resource pool;
获取模块403,利用存储资源池获取待存储数据的多个副本;The acquisition module 403 uses the storage resource pool to acquire multiple copies of the data to be stored;
存储模块404,利用容器从节点将数据标识和待存储数据的副本存储在目标数据子节点上。The storage module 404 uses the container slave node to store the data identifier and a copy of the data to be stored on the target data sub-node.
在本发明实施例中,存储模块404,还用于:利用容器从节点将数据标识和待存储数据的副本存储在目标数据子节点上之后,利用容器从节点将数据标识、待存储数据的副本和目标数据子节点的对应关系存储在元数据节点上。In the embodiment of the present invention, the storage module 404 is also used to: after using the container slave node to store the data identifier and the copy of the data to be stored on the target data sub-node, use the container slave node to store the data identifier and the copy of the data to be stored. The corresponding relationship with the target data sub-node is stored on the metadata node.
在本发明实施例中,该装置还包括查询模块,用于:利用容器主节点接收客户端发送的数据查询请求,数据查询请求指示了数据标识;利用存储主节点确定与数据查询请求对应的目标元数据节点;利用存储资源池根据目标元数据节点确定与数据标识对应的多个目标数据子节点;利用容器从节点从多个目标数据子节点中获取与数据查询请求对应的查询结果。In the embodiment of the present invention, the device also includes a query module, which is used to: use the container master node to receive the data query request sent by the client, where the data query request indicates the data identifier; use the storage master node to determine the target corresponding to the data query request Metadata node; use the storage resource pool to determine multiple target data sub-nodes corresponding to the data identifier according to the target metadata node; use the container slave node to obtain query results corresponding to the data query request from multiple target data sub-nodes.
根据本发明实施例的另一个方面,提供了一种电子设备,包括:一个或多个处理器;存储装置,用于存储一个或多个程序,当一个或多个程序被一个或多个处理器执行,使得一个或多个处理器实现本发明提供的基于云原生平台系统的数据处理的方法。According to another aspect of an embodiment of the present invention, an electronic device is provided, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are processed by one or more The processor executes, causing one or more processors to implement the data processing method based on the cloud native platform system provided by the present invention.
根据本发明实施例的还一个方面,提供了一种计算机可读介质,其上存储有计算机程序,程序被处理器执行时实现本发明提供的基于云原生平台系统的数据处理的方法。According to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method based on the cloud native platform system provided by the present invention is implemented.
图5示出了可以应用本发明实施例的基于云原生平台系统的数据处理的方法或基于云原生平台系统的数据处理的装置的示例性系统架构500。FIG. 5 shows an exemplary system architecture 500 in which the data processing method based on the cloud native platform system or the data processing device based on the cloud native platform system according to the embodiment of the present invention can be applied.
如图5所示,系统架构500可以包括终端设备501、502、503,网络504和服务器505。网络504用以在终端设备501、502、503和服务器505之间提供通信链路的介质。网络504可以包括各种连接类型,例如有线、无线通信链路或者光纤电缆等等。As shown in Figure 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
用户可以使用终端设备501、502、503通过网络504与服务器505交互,以接收或发送消息等。终端设备501、502、503上可以安装有各种通讯客户端应用,例如购物类应用、网页浏览器应用、搜索类应用、即时通信工具、邮箱客户端、社交平台软件等(仅为示例)。Users can use terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).
终端设备501、502、503可以是具有显示屏并且支持网页浏览的各种电子设备,包括但不限于智能手机、平板电脑、膝上型便携计算机和台式计算机等等。The terminal devices 501, 502, and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and so on.
服务器505可以是提供各种服务的服务器,例如对用户利用终端设备501、502、503所浏览的购物类网站提供支持的后台管理服务器(仅为示例)。后台管理服务器可以对接收到的产品信息查询请求等数据进行分析等处理,并将处理结果(例如目标推送信息、产品信息--仅为示例)反馈给终端设备。The server 505 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503 (example only). The background management server can analyze and process the received product information query request and other data, and feed back the processing results (such as target push information, product information - only examples) to the terminal device.
需要说明的是,本发明实施例所提供的基于云原生平台系统的数据处理的方法一般由服务器505执行,相应地,基于云原生平台系统的数据处理的装置一般设置于服务器505中。It should be noted that the data processing method based on the cloud native platform system provided by the embodiment of the present invention is generally executed by the server 505. Correspondingly, the data processing device based on the cloud native platform system is generally provided in the server 505.
应该理解,图5中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要,可以具有任意数目的终端设备、网络和服务器。It should be understood that the number of terminal devices, networks and servers in Figure 5 is only illustrative. Depending on implementation needs, there can be any number of end devices, networks, and servers.
下面参考图6,其示出了适于用来实现本发明实施例的终端设备的计算机系统600的结构示意图。图6示出的终端设备仅仅是一个示例,不应对本发明实施例的功能和使用范围带来任何限制。Referring now to FIG. 6 , which shows a schematic structural diagram of a computer system 600 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 6 is only an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present invention.
如图6所示,计算机系统600包括中央处理单元(CPU)601,其可以根据存储在只读存储器(ROM)602中的程序或者从存储部分608加载到随机访问存储器(RAM)603中的程序而执行各种适当的动作和处理。在RAM 603中,还存储有系统600操作所需的各种程序和数据。CPU 601、ROM 602以及RAM 603通过总线604彼此相连。输入/输出(I/O)接口605也连接至总线604。As shown in FIG. 6 , computer system 600 includes a central processing unit (CPU) 601 that can operate according to a program stored in a read-only memory (ROM) 602 or loaded from a storage portion 608 into a random access memory (RAM) 603 And perform various appropriate actions and processing. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
以下部件连接至I/O接口605:包括键盘、鼠标等的输入部分606;包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分607;包括硬盘等的存储部分608;以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分609。通信部分609经由诸如因特网的网络执行通信处理。驱动器610也根据需要连接至I/O接口605。可拆卸介质611,诸如磁盘、光盘、磁光盘、半导体存储器等等,根据需要安装在驱动器610上,以便于从其上读出的计算机程序根据需要被安装入存储部分608。The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., speakers, etc.; and a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. Driver 610 is also connected to I/O interface 605 as needed. Removable media 611, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.
特别地,根据本发明公开的实施例,上文参考流程图描述的过程可以被实现为计算机软件程序。例如,本发明公开的实施例包括一种计算机程序产品,其包括承载在计算机可读介质上的计算机程序,该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中,该计算机程序可以通过通信部分609从网络上被下载和安装,和/或从可拆卸介质611被安装。在该计算机程序被中央处理单元(CPU)601执行时,执行本发明的系统中限定的上述功能。In particular, according to embodiments disclosed in the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the disclosed embodiments of the present invention include a computer program product including a computer program carried on a computer-readable medium, the computer program including program code for executing the method shown in the flowchart. In such embodiments, the computer program may be downloaded and installed from the network via communication portion 609, and/or installed from removable media 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are performed.
需要说明的是,本发明所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件,或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于:具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本发明中,计算机可读存储介质可以是任何包含或存储程序的有形介质,该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本发明中,计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号,其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式,包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质,该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输,包括但不限于:无线、电线、光缆、RF等等,或者上述的任意合适的组合。It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmed read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.
附图中的流程图和框图,图示了按照本发明各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分,上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意,在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个接连地表示的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图或流程图中的每个方框、以及框图或流程图中的方框的组合,可以用执行规定的功能或操作的专用的基于硬件的系统来实现,或者可以用专用硬件与计算机指令的组合来实现。The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagram or flowchart illustration, and combinations of blocks in the block diagram or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or may be implemented by special purpose hardware-based systems that perform the specified functions or operations. Achieved by a combination of specialized hardware and computer instructions.
描述于本发明实施例中所涉及到的模块可以通过软件的方式实现,也可以通过硬件的方式来实现。所描述的模块也可以设置在处理器中,例如,可以描述为:一种处理器包括接收模块、确定模块、获取模块和存储模块。其中,这些模块的名称在某种情况下并不构成对该模块本身的限定,例如,接收模块还可以被描述为“利用云原生平台系统的容器主节点接收客户端发送的数据存储请求的模块”。The modules involved in the embodiments of the present invention can be implemented in software or hardware. The described module can also be provided in a processor. For example, it can be described as: a processor includes a receiving module, a determining module, an obtaining module and a storage module. Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances. For example, the receiving module can also be described as "a module that uses the container master node of the cloud native platform system to receive data storage requests sent by the client. ".
作为另一方面,本发明还提供了一种计算机可读介质,该计算机可读介质可以是上述实施例中描述的设备中所包含的;也可以是单独存在,而未装配入该设备中。上述计算机可读介质承载有一个或者多个程序,当上述一个或者多个程序被一个该设备执行时,使得该设备包括:利用云原生平台系统的容器主节点接收客户端发送的数据存储请求,数据存储请求包括数据标识和与数据标识对应的待存储数据;利用存储主节点根据存储资源池确定与待存储数据对应的多个目标数据子节点;利用存储资源池获取待存储数据的多个副本;利用容器从节点将数据标识和待存储数据的副本存储在目标数据子节点上。As another aspect, the present invention also provides a computer-readable medium. The computer-readable medium may be included in the device described in the above embodiments; it may also exist separately without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: using the container master node of the cloud native platform system to receive the data storage request sent by the client, The data storage request includes a data identifier and the data to be stored corresponding to the data identifier; the storage master node is used to determine multiple target data sub-nodes corresponding to the data to be stored according to the storage resource pool; the storage resource pool is used to obtain multiple copies of the data to be stored ;Use the container slave node to store the data identification and a copy of the data to be stored on the target data child node.
根据本发明实施例的技术方案,本发明实施例提供的云原生平台系统以及基于云原生平台系统的数据处理方法,该云原生平台系统通过将分布式存储系统的组件容器化,将容器集群与分布式存储系统混合部署,充分利用容器集群中本地硬盘的存储能力,克服了容器数据持久化依赖外部存储的问题,并实现了应用的数据存储和计算的分离,降低了存储成本和运维复杂度;分布式存储的存储资源池可以作为共享存储用于容器集群的持久化存储,形成融合的云原生容器平台;该数据处理的方法能够实现数据的多副本保护,提高数据可靠性,且能够有效避免跨网络数据访问造成的开销和延迟。According to the technical solution of the embodiment of the present invention, the cloud native platform system and the data processing method based on the cloud native platform system are provided by the embodiment of the present invention. The cloud native platform system combines the container cluster and the data processing method by containerizing the components of the distributed storage system. The hybrid deployment of distributed storage systems makes full use of the storage capabilities of local hard disks in container clusters, overcomes the problem of container data persistence relying on external storage, and realizes the separation of application data storage and computing, reducing storage costs and operation and maintenance complexity. degree; the storage resource pool of distributed storage can be used as shared storage for persistent storage of container clusters, forming a converged cloud-native container platform; this data processing method can achieve multiple copies of data protection, improve data reliability, and can Effectively avoid overhead and delays caused by cross-network data access.
上述具体实施方式,并不构成对本发明保护范围的限制。本领域技术人员应该明白的是,取决于设计要求和其他因素,可以发生各种各样的修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等,均应包含在本发明保护范围之内。The above-mentioned specific embodiments do not constitute a limitation on the scope of the present invention. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311586712.1A CN117499396A (en) | 2023-11-24 | 2023-11-24 | Data processing method and device based on cloud native platform system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311586712.1A CN117499396A (en) | 2023-11-24 | 2023-11-24 | Data processing method and device based on cloud native platform system |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117499396A true CN117499396A (en) | 2024-02-02 |
Family
ID=89682844
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311586712.1A Pending CN117499396A (en) | 2023-11-24 | 2023-11-24 | Data processing method and device based on cloud native platform system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117499396A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118916184A (en) * | 2024-10-11 | 2024-11-08 | 北京新里程叮铃科技有限公司 | Service development method, device and computer program product |
-
2023
- 2023-11-24 CN CN202311586712.1A patent/CN117499396A/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118916184A (en) * | 2024-10-11 | 2024-11-08 | 北京新里程叮铃科技有限公司 | Service development method, device and computer program product |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111045833B (en) | Interface calling method and device | |
CN112612467B (en) | A method and device for processing a micro front-end architecture based on Qiankun | |
US8266630B2 (en) | High-performance XML processing in a common event infrastructure | |
JP7081014B2 (en) | Methods and devices for adjusting the number of instances, electronic devices, storage media and computer programs | |
US11880727B2 (en) | Dynamic application programming interface (API) contract generation and conversion through microservice sidecars | |
CN110162410B (en) | A message processing method and device | |
CN105554085A (en) | Server connection-based dynamic timeout processing method and device | |
CN114625479A (en) | Cloud edge collaborative application management method in edge computing and corresponding device | |
CN117499396A (en) | Data processing method and device based on cloud native platform system | |
CN112905273A (en) | Service calling method and device | |
CN112306695A (en) | Data processing method and device, electronic equipment and computer storage medium | |
JP2014106728A (en) | Multi-tenant type service system, information processing device, control method, and program | |
WO2023082716A1 (en) | Method and apparatus for operating android application in linux system, and device | |
US20090132582A1 (en) | Processor-server hybrid system for processing data | |
CN115562887A (en) | Inter-core data communication method, system, device and medium based on data package | |
Indrasiri | Introduction to WSO2 ESB | |
US11683400B1 (en) | Communication protocol for Knative Eventing's Kafka components | |
CN114896244A (en) | Method, apparatus, apparatus, and computer-readable medium for configuring database tables | |
CN114356873A (en) | Data sharing system and method | |
CN115562871A (en) | Memory allocation management method and device | |
CN110661857B (en) | Data synchronization method and device | |
US10817334B1 (en) | Real-time analysis of data streaming objects for distributed stream processing | |
CN111970338A (en) | Request processing method and device based on cloud function and computer readable medium | |
US12197461B2 (en) | Distributed function data transformation system | |
CN113132480B (en) | Data transmission method, device and system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |