TWI695329B - Data sharding management system and method on container platform - Google Patents

Abstract

This invention provides a data sharding management system and a method on a container platform. The system is based on the original container management mechanism of the container platform, and incorporates a data sharding management mechanism to enable the container platform to natively support functions such as data partitioning of large data applications, separation of master-slave, and the like, and dispersedly store the data in the storage medium that comes with the host group where the container platform is located, such that the container platform does not need to build additional distributed storage media, and does not need to rely on the data sharding management mechanism of an individual distributed data management software.

Description

Data fragment management system and method built on container platform

The invention relates to a data fragment management system and method, in particular to a data fragment management system built on a container platform and a method for implementing data fragment management.

With the popularization of container applications, container platforms have gradually entered the enterprise's infrastructure stack. However, conventional container platforms are good at supporting the deployment and expansion of stateless applications, but for stateful ( Stateful) application support is still not as complete as stateless applications.

It is known that if the current container platform needs to store permanent status data, it needs to rely on the external storage medium. However, the external storage medium is expensive to expand, and it is difficult to store large amounts of data. In addition, for large data applications, the data is divided into fragments and distributed to save costs.

For example, a general-purpose container platform, such as Kubernetes, does not have built-in storage media management. It requires an external storage media management system to store persistent state data. The conventional container platform cannot manage large data applications due to lack of management to provide storage media, and needs to rely on big data application software to provide decentralized data management 理 Functionality.

In order to solve the above problems, this case discloses a data fragmentation management system on the container platform. The system is based on the container management mechanism of the container platform, plus the data fragmentation management mechanism, so that the container platform can natively support data segmentation of large data applications, Functions such as master-slave separation, and distributed storage of data in the storage media of the host group where the container platform is located, without the need to build additional distributed storage media or rely on distributed data management software.

First, the data fragment management system on the container platform of this case includes: a coordinator for accessing cluster information; a plurality of computing nodes for deploying containers on each computing node to access the cluster information; a plurality of memories , Each of which is connected to at least one of the plurality of computing nodes; a container collaboration module, used to receive a request to set up a data service and access the cluster information, wherein the data service is divided into a plurality of data fragments, and according to the The requirements for setting up data services and the cluster information determine the memory used to store each data fragment and the computing node where each container used to serve each data fragment is located. Update the cluster information on the decision result of the computing node for each computing node to open a corresponding number of containers according to the updated cluster information, and the coordinator further coordinates at least one when the opened container is registered with the coordinator A leader container; and a load balancer for receiving user data requests for the data service and accessing the cluster information to direct the data requests to the agent components of the computing nodes according to the updated cluster information, and The agent component further directs the data request to the container in each computing node.

Secondly, the management method of data fragments on the container platform in this case includes: The request to set up a data service, where the data service is divided into a plurality of data fragments; according to the request for the set up data service and a cluster of information, the memory used to store each data fragment is determined, and the data is used to serve each data The computing node where each container of the shard is located; update the cluster information according to the decision result of the memory and the computing node; open the corresponding number of containers according to the updated cluster information to update the access address of the successfully opened container To the cluster information; register each of the containers to coordinate at least one leader container from the container serving the same data fragment, to write the coordination result into the cluster information, and then complete the request for setting up the data service; and receive and use For the data request of the completed data service, the data request is directed to the agent component in the computing node according to the updated cluster information, and the data request is further directed to the container in the computing node.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below in conjunction with the accompanying drawings for detailed description. Additional features and advantages of the present invention will be partially explained in the following description, and these features and advantages will be partially known from the description, or may be learned by practicing the present invention. The features and advantages of the present invention are recognized and achieved by means of the elements and combinations particularly pointed out in the scope of the patent application. It should be understood that both the foregoing general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the scope of the claimed invention.

100‧‧‧Container Collaboration Module

200, 300‧‧‧ cluster

210, 220, 310‧‧‧ computing nodes

211, 221, 311

212, 222, 312

213, 223, 313, 314‧‧‧ container

230, 320‧‧‧ memory

400‧‧‧Coordinator

500‧‧‧Load balancer

610~613, 615, 620~622, 630~633 ‧‧‧ data block

614, 615, 623

S101~S105, S201~S204 ‧‧‧ steps

Figure 1 is a system architecture diagram showing a data fragment management system on a container platform according to the specific embodiment disclosed in this case.

Figure 2 shows the information on the container platform according to the specific embodiment disclosed in this case. A data structure diagram of cluster information managed by a material fragment management system.

Figure 3 is a diagram showing a routing setting for a data fragment management system on a container platform to receive user data requests according to a specific embodiment disclosed in this case.

Figure 4 is a flowchart showing the control process of managing data fragments according to the specific embodiment disclosed in this case.

Figure 5 is a flowchart showing a request process for managing data fragments according to the specific embodiment disclosed in this case.

What is disclosed in this article is a data fragment management system and method on a container platform.

Please refer to the drawings. Figure 1 shows the system architecture of the data fragment management system on the container platform. Computing nodes (eg, hosts) 210, 220, and 310 can open respective containers 213, 223, 313, and 314. Each of the memories (eg, storage media) 230 and 320 may be connected to at least one of the computing nodes 210, 220, and 310. The coordinator 400 accesses cluster information. A container collaboration module (such as Container Orchestration Tool) 100 is used to receive a request for an erection data service, where the data service is divided into restatement data fragments to comply with the requirements for the erection data service and the cluster information, Determine the computing node 210, 220 or 310 where the memory 230 or 320 for storing each data fragment and the container 213, 223 or 313 for serving each data fragment are located. In addition, the load balancer 500 can receive data requests from users for the data service.

In the system architecture shown, the function of the container collaboration module 100 is to manage the computing node (Node). In the specific embodiment disclosed in this case, the container collaboration module 100 manages A container platform composed of computing nodes 210 (Node1), 220 (Node2), 310 (Node3), wherein, in this embodiment, the computing nodes 210, 220 belong to the cluster 200 (Cluster1) and share storage (Storage) 230, and computing Node 310 belongs to cluster 300 (Cluster2) and uses storage 320.

The container collaboration module 100 disclosed in this case has an additional function for receiving a data service provider's request for setting up a data service, so as to convert the request into cluster information (Clusters, as shown in Fig. 2), and transfer this The cluster information is stored in Coordinator 400.

The computing nodes 210, 220, and 310 may include multiple activated containers. For example, in the specific embodiment disclosed in this case, the container 213 is activated in the computing node 210, the container 223 is activated in the computing node 220, and the containers 313 and 314 are activated in the computing node 310.

The computing nodes 210, 220, 310 also include control components 211, 221, 311 and agent components 212, 222, 312 that exist in a resident manner.

The functions of the control components 211, 221, and 311 are to manage containers for the operation nodes to which they belong, and to subscribe to the cluster information stored in the coordinator 400 (as shown in FIG. 2), and add or delete containers based on the cluster information. When it fails, a new container is started to replace it, and the managed container status is reported to the coordinator 400 to update the cluster information.

The function of the proxy component 212, 222, 312 is to receive the user request imported by the load balancer 500 and direct the request to the container in the computing node according to the appropriate routing table in the cache information stored in it. The proxy components 212, 222, and 312 disclosed in this case also have an additional function: subscribing to the cluster information in the coordinator 400 (as shown in FIG. 2) to obtain a routing table for each data fragment in the cluster information.

The function of the load balancer 500 is to accept user data read and write requests, and According to the cluster information in the cache information it stores, the routing method for guiding the data read-write request is determined. The cluster information is obtained from the cluster information in the subscription coordinator 400 (as shown in FIG. 2).

The function of the coordinator 400 is to maintain the global cluster information of the data fragmentation management system (as shown in Figure 2). The cluster information records the following metadata (including metadata), including but not limited to: the control components of the respective computing nodes in the cluster and The access address of the agent component, the container started in each computing node, the data fragments and service types (for example, read/write) that each container can serve, and the method of data slicing. The above metadata all provide a subscription function and notify the subscriber to obtain new data when the metadata is updated. The above metadata is provided by the container collaboration module 100 and the control components 211, 221, and 311, and is subscribed by the control components 211, 221, and 311, the proxy components 212, 222, and 312, and the load balancer 500.

Another function of the coordinator 400 is to provide Leader Election. The leader election will coordinate multiple potential leaders in the container opened by the data fragment management system to achieve a decentralized consensus and choose one of them. leader.

Please refer to FIG. 2 to reveal the cluster information (Clusters) maintained by the coordinator 400 in the specific embodiment of the present case. The cluster information is stored in a tree-like manner, wherein the cluster information is received by the data service provider to set up a data service request The initial state before is as follows.

As described above, the container collaboration module 100 manages the cluster 200 (Cluster1) and the cluster 300 (Cluster2). The cluster 200 includes a computing node 210 (Node1), a computing node 220 (Node2), and a storage 230 (Storage). The states described above are based on the data block 611 and the data subdirectory 614 under the data block 610 in FIG. 2. The data block 612, the data subdirectory 615 and the data block 613 are represented. The cluster 300 includes a computing node 310 (Node3) and a memory 320, the state described above It is represented by data block 621, data subdirectory 623, and data block 622 under data block 620 in Figure 2.

The cluster information of the coordinator 400 also stores the access addresses of the control component and the proxy component in each computing node. For example, the data block 611 stores the access addresses of the control component and the proxy component of Node1; the data block 612 stores the node 2 The access addresses of the control component and the proxy component; and the data block 621 stores the access addresses of the control component and the proxy component of Node3.

The cluster information of the coordinator 400 further records the data services that each computing node can provide and stores them in the data subdirectory. For example, the data subdirectory 614 records data services that Node1 can provide; the data subdirectory 615 records data services that Node2 can provide; and the data subdirectory 623 records data services that Node3 can provide. It is worth noting that the data subdirectories 614, 615, 623 are all empty in the cluster information in the initial state, and there is no data yet. And the materials in the material sub-directories 614, 615, 623 are subscribed by the control members 211, 221, 311 of the corresponding computing nodes, respectively.

The cluster information of the coordinator 400 also records the access address of the memory used by each cluster in the data fragment management system. For example, the data block 613 stores the access address of the memory 230 used by Cluster1; and the data block 622 stores the access address of the memory 320 used by Cluster2.

The method for managing data fragments disclosed in this case is divided into two parts: Control Flow and Request Flow, which are described on the basis of Figures 1 and 2 based on Figure 4 and Figure 5, respectively.

Figure 4 reveals the flow chart of the control flow. Please refer to step S101. The control flow starts when the container collaboration module 100 receives the request of the data service provider to set up a data service. In the specific embodiment disclosed in the case, the data service that the data service provider wants to set up is a key-value data service, which is named Data1. In addition, Data1 specifications required by data service providers are as follows:

-The primary key range of Data1 is 0000 to 9999

-Data1 is divided into two data fragments, the primary key range of the first data fragment is 0000 to 4999, and the primary key range of the second data fragment is 5000 to 9999

-Each data fragment needs two containers to provide services, one of which can provide write/read services, and the other can provide read services.

Referring to step S102, the container collaboration module 100 selects a memory as a storage carrier for each data fragment according to the above-required Data1 specifications and the cluster information currently in the initial state in the coordinator 400. Assuming that the current selection method is Round-Robin, the memory 230 of the cluster 200 (Cluster1) is selected as the storage carrier of the first data fragment (Shard1), and the memory 320 of the cluster 300 (Cluster2) is selected as the second data Storage carrier for Shard2. In addition, other selection methods can also select a computing node that is currently less burdened, or deploy multiple (for example, two) data services on the same computing node according to application requirements.

The container collaboration module 100 must also select the computing node where the container opened for serving the data fragments according to the required Data1 specification. Among them, if the selected storage carrier is connected by multiple computing nodes, it is also assumed that the computing nodes are selected on a round-robin basis. As in the specific embodiment of this case, the memory 230 of the cluster 200 (Cluster1) is connected by the computing nodes 210 (Node1), 220 (Node2), so the container collaboration module 100 will choose to open a container on the computing node 210 (Node1) to serve The first data fragment, and another container is opened on the computing node 220 (Node2) to serve the first data fragment.

Since the memory 320 of the cluster 300 (Cluster2) is only connected by the computing node 310 (Node3), the container collaboration module 100 will select to open two containers on the computing node 310 (Node3) to serve the second data fragment.

Referring to step S103, the container collaboration module 100 writes the above selection results of the memory and computing nodes into the data blocks 610 and 620 in the cluster information of the coordinator 400 (as shown in FIG. 2), that is, Write a "Data1/Shard1/1" to Node1's data subdirectory 614; write a "Data1/Shard1/2" to Node2's data subdirectory 615; and write a "Data2/Shard2/3" and another "Data2" /Shard2/4" is written to Node3's data subdirectory 623, respectively. It is worth noting that the Master and Slave shown in the data subdirectories 614, 615, and 623 in Figure 2 need to be updated in subsequent steps; the container access address (IP) must be updated by the control component. Not yet written.

The container collaboration module 100 will also add a data block 630 to the cluster information of the coordinator 400 to store the service address of the data service Data1 and metadata describing the specifications of each data fragment.

For example, in the embodiment disclosed in this case, the container collaboration module 100 will generate the service address of the data service Data1 (for example: 10.0.0.1:6379) and record it in the data block 631. The user can use this service address to store Get data service Data1.

The container collaboration module 100 also stores the metadata describing the specifications of each data fragment requested by the data service provider in the data block 630. For example, the data block 632 records the primary key range of the first data shard (Shard1) as 0000-4999, and records the members of the first data shard as Node1 and Node2 under the member subdirectory (Member). The first piece of information learned that Node1 will have a master container (Master); and the data block 633 will record the second data The primary key range of Shard2 is 5000-9999, and the member of the second service data shard recorded in the member subdirectory (Member) is Node3.

The above are the actions performed by the container collaboration module 100 in response to the request of the data service provider to set up the data service Data1.

Referring to step S104, the control components 211, 221, and 311 are subscribed to the data subdirectories 614, 615, and 623 stored in the coordinator 400, respectively. Therefore, when the container collaboration module 100 writes data, the notification is obtained and the content is subscribed according to the subscription content. Open the corresponding container for the newly added data in the data sub-directories 614, 615, and 623. That is, the control component 211 opens the container 213 for the newly added "Data1/Shard1/1" in the data subdirectory 614, and updates its access address (for example, 192.168.1.2:1111) to the data subdirectory 614; The control component 221 opens the container 223 for the newly added "Data1/Shard1/2" in the data subdirectory 615, and updates its access address (for example, 192.168.1.2:1111) to the data subdirectory 615; and the control component 311 opens the container 223 for the newly added "Data2/Shard2/3" and "Data2/Shard2/4" in the data subdirectory 623, and accesses the addresses (for example, 192.168.1.2: 1111 and 192.168.1.3: 1111 ) Update to the information subdirectory 623.

The control components 211, 221, and 311 use the built-in health check mechanism of the container platform to regularly check the health status of the containers they manage, delete unhealthy containers, and replace them with new ones. 400 Update the container access address recorded in the data sub-directories 614, 615, 623.

The above is the operation performed by the control members 211, 221, and 311.

Please refer to step S105. When the container is successfully started, it will submit a registration to the coordinator 400 and trigger a leader election (Leader Election) mechanism. The details are as follows.

The coordinator 400 disclosed in this case is based on etcd (https://coreos.com/etcd/) or Zookeeper (https://zookeeper.apache.org/) implementation, both of which support hierarchical (and tree-like) data organization, can be used to store the content of the data cluster as shown in Figure 2, and support leader election The mechanism can allow multiple itineraries to register with a directory, coordinate the multiple itineraries, and select one itinerary as the leader. In the specific embodiment disclosed in this case, the containers 213 and 223 (ie the itinerary) are registered with the data block 632 (ie the directory) of the coordinator 400. If the leader is still not selected, the leader election mechanism will be used to A leader is coordinated in the containers 213, 223. In this embodiment, the container 213 is the selected leader; and, the containers 313, 314 register with the data block 633 of the coordinator 400. If the leader is not yet selected, A leader is selected from the containers 313 and 314 through a leader election mechanism. In this embodiment, the container 314 is the selected leader. The leader described here is the master container in the data service, and the remaining containers are slaves. The result of this leader election will be written into the sub-directories 614, 615, 623 of the data stored in the coordinator 400 and marked as "Master" or "Slave".

At this point, the requirements for the data service provider to set up the data service Data1 have been completed, and the master/slave container service of the second data fragment has been completed. It can be prepared to provide users with requests. Please refer to the use flow shown in Figure 5 (Request Flow ).

Please refer to step S201. The user's request for the data service Data1 can be specified by the REST Protocol. For example, if you want to write a piece of data with a primary key of 4999 in Data1, you can issue an HTTP POST request (for example: http://1/0.0.0.1: 6379/4999), and write the value to be written in POST Data for transmission. If you want to read a piece of data with a primary key of 4999 in Data1, you can send an HTTP GET request (for example: http://1/0.0.0.1: 6379/4999).

Referring to steps S202 and S203, the user's request is sent to the load balancer 500. The load balancer 500 subscribes to all cluster information maintained by the coordinator 400, so it can directly participate in According to the cache information of the cluster information stored locally, the route to guide the user's request is determined. The specific embodiment disclosed in this case uses HAProxy as the load balancer, and its routing settings can be seen in FIG. 3.

As shown in the specific embodiment of this case, when the load balancer 500 receives an HTTP POST request (for example, http://10.0.0.1:6379/4999), the load balancer 500 can use its local cache storage coordinator 400 The saved data block 630 determines that the data requested by the user is located in the first data fragment (ie, Data1/Shard1), and because the request requires writing to the service, it can be inferred that the request needs to be directed to the service first data According to the data block 632, the master container of the shard Shard1 can be known as Node1, so the request is directed to the proxy component 212 of the computing node Node1 according to the container access address stored in the data block 614.

Referring to step S204, the proxy component on each computing node also subscribes to the metadata of the computing node it belongs to in the coordinator 400. For example, the proxy component 212 subscribes to the data block 611 and the data subdirectory 614, and stores it on the proxy component 212 Cached information of the subscribed data block 611 and data subdirectory 614. When the proxy component receives the HTTP POST request (http://10.0.0.1:6379/4999) from the load balancer 500, it can use this cache information to rewrite the request as http://192.168.1.2:1111 /4999and leads to the container 213.

The above are the steps of using the process.

The above-mentioned embodiments only exemplarily illustrate the principles, characteristics and effects of the present invention, and are not intended to limit the scope of the invention. Anyone who is familiar with this skill can do the above without departing from the spirit and scope of the present invention. The embodiment is modified and changed. Any equivalent changes and modifications made using the disclosure of the present invention should still be covered by the scope of the patent application. Therefore, the scope of protection of the rights of the present invention should be as listed in the scope of patent application.

100‧‧‧Container Collaboration Module

200, 300‧‧‧ cluster

210, 220, 310‧‧‧ computing nodes

211, 221, 311

212, 222, 312

213, 223, 313, 314‧‧‧ container

230, 320‧‧‧ memory

400‧‧‧Coordinator

500‧‧‧Load balancer

Claims (12)

A data fragment management system on a container platform, including: a coordinator for managing cluster information; a plurality of computing nodes for deploying containers in each computing node to access the cluster information; a plurality of memories, each of which Connected to at least one of the plurality of computing nodes; a container collaboration module for receiving a request for setting up a data service and accessing the cluster information, wherein the data service is divided into a plurality of data fragments to conform to the setting up data service The requirements and the cluster information determine the memory used to store each data fragment and the computing node where each container serving each data fragment is located, wherein the container coordination module is based on the memory and the computing node The decision result updates the cluster information for each computing node to open a corresponding number of containers according to the updated cluster information, and the coordinator further coordinates at least one leader container when the opened container registers with the coordinator; And a load balancer for receiving data requests for the data service and accessing the cluster information. The data fragment management system as described in item 1 of the patent scope, wherein the container collaboration module writes the decision results of the memory and the computing node into the data subdirectory of the data cluster, and the information in the cluster A data block is newly added to store the service address of the data service and metadata of each data fragment. The data fragment management system as described in item 1 of the patent application scope, wherein the cluster information includes the access addresses of the proxy components and control components of the plurality of computing nodes, the access addresses of the memory, and the data service Service address and metadata of each piece of data. The data fragment management system as described in item 1 of the patent application scope, wherein each of the computing nodes includes its own control component for subscribing to the cluster information of the coordinator, opening or deleting containers based on the cluster information, and Report the status of each container to the coordinator. The data fragment management system as described in item 1 of the patent application scope, wherein the load balancer further subscribes to the cluster information of the coordinator to decide to direct the data request to the proxy components of the plurality of computing nodes according to the cluster information Route. The data fragment management system as described in item 1 of the patent application scope, wherein each of the computing nodes includes its own proxy component, which is used to subscribe to the cluster information and receive data requests from the load balancer to transfer The data request is directed to each container. A method for managing data fragments on a container platform includes: causing a container collaboration module to receive a request to erect data services, wherein the data service is divided into a plurality of data fragments; and the container collaboration module is to comply with the requirements for erecting data services And a cluster of information, determine the memory used to store each of the data fragments, and determine the computing node where each container serving each of the data fragments is located; make the container cooperative module based on the decision on the memory and the computing node The result updates the cluster information; causes the computing node to open a corresponding number of containers according to the updated cluster information, so as to update the access address of the successfully opened container to the cluster information; Let the coordinator register each of the containers to coordinate at least one leader container from the container serving the same data fragment to write the coordination result into the cluster information to complete the request for setting up the data service; and make the load balancer receive Data request for the completed data service. The method as described in item 7 of the patent application scope, wherein, when the cluster information is in an initial state, the cluster information includes the access addresses of the control component and the proxy component of the computing node, and the access address of the memory , And the data subdirectory of the cluster information is empty. The method according to item 7 of the patent application scope, wherein the command to update the cluster information based on the determination result of the memory and the computing node of the container coordination module includes: causing the container cooperation module to update the memory And the decision result of the computing node is written into the data subdirectory of the cluster information; and the container cooperative module adds a data block to the cluster information to store the service address of the data service and each data fragment Metadata. The method as described in item 7 of the patent application scope further includes causing the control component of the computing node to periodically check the health status of the container in the computing node through a health check mechanism. The method as described in item 7 of the patent application scope, wherein the causing the load balancer to receive the data request for the completed data service further includes: the user accessing the data service using the service address of the data service; The user specifies the data request for the data service; causes the load balancer to receive the data request and decides the routing of the agent component that directs the data request to the computing node; and causes the agent component to direct the data request to the data service The container in the operation node. The method described in item 7 of the patent application scope, wherein the container coordination module determines that the computing node where each container used to serve each data fragment is located is selected by a round-robin system, or Choose a computing node that is currently less burdensome, or deploy multiple data services on the same computing node according to the needs of the application.

Images