CN118158222A - Load balancer deployment method, device, electronic equipment, storage medium and product - Google Patents

Load balancer deployment method, device, electronic equipment, storage medium and product Download PDF

Info

Publication number
CN118158222A
CN118158222A CN202410581669.8A CN202410581669A CN118158222A CN 118158222 A CN118158222 A CN 118158222A CN 202410581669 A CN202410581669 A CN 202410581669A CN 118158222 A CN118158222 A CN 118158222A
Authority
CN
China
Prior art keywords
cluster
instance
subnet
target
cloud platform
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410581669.8A
Other languages
Chinese (zh)
Other versions
CN118158222B (en
Inventor
陶慧
李维亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
China Mobile Suzhou Software Technology Co Ltd
Original Assignee
China Mobile Communications Group Co Ltd
China Mobile Suzhou Software Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd, China Mobile Suzhou Software Technology Co Ltd filed Critical China Mobile Communications Group Co Ltd
Priority to CN202410581669.8A priority Critical patent/CN118158222B/en
Publication of CN118158222A publication Critical patent/CN118158222A/en
Application granted granted Critical
Publication of CN118158222B publication Critical patent/CN118158222B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/10015—Access to distributed or replicated servers, e.g. using brokers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/30—Profiles
    • H04L67/303—Terminal profiles

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The disclosure provides a load balancer deployment method, a load balancer deployment device, electronic equipment, a storage medium and a load balancer product. The method comprises the following steps: acquiring cluster information of each cluster in a target cloud platform, wherein the cluster information at least comprises a corresponding load balancer LB instance deployed by the cluster, a subnet to which the LB instance belongs and a cluster resource utilization rate; responding to a request for creating a target LB instance belonging to a first subnet, and judging whether the LB instance belonging to the first subnet exists in the cluster of the target cloud platform; if the judgment result is negative, the target LB instance is deployed on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform.

Description

Load balancer deployment method, device, electronic equipment, storage medium and product
Technical Field
The disclosure relates to the technical field of cloud computing, and in particular relates to a load balancer deployment method, a load balancer deployment device, electronic equipment, a storage medium and a product.
Background
Load balancing is a key component of a high-availability network infrastructure, cloud platform users can typically deploy multiple application servers, different application servers process different services, and then distribute requests to different servers in a cluster by configuring Load Balancing (LB) instances in the cluster, thereby improving performance and reliability of websites, applications, databases, or other services.
However, in the related art, each time an LB instance is created, a plurality of Message Queue (MQ) messages are required to be sent to query the resource usage situation on each cluster, which reduces the availability of the clusters, and if the MQ service has a problem, the usage situation of the cluster resources cannot be obtained, resulting in failure of creating the LB instance in the clusters.
Disclosure of Invention
The present disclosure provides a load balancer deployment method, apparatus, electronic device, storage medium, and product to solve the problems in the related art.
An embodiment of a first aspect of the present disclosure provides a load balancer deployment method, including:
acquiring cluster information of each cluster in a target cloud platform, wherein the cluster information at least comprises a load balancer LB instance, a subnet to which the LB instance belongs and a cluster resource utilization rate of which the corresponding cluster is deployed;
Responding to a request for creating a target LB instance belonging to a first subnet, and judging whether the LB instance belonging to the first subnet exists in a cluster of a target cloud platform;
if the judgment result is negative, the target LB instance is deployed on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform.
In an embodiment, after determining whether the LB instance belonging to the first subnet exists in the cluster of the target cloud platform, the method provided by the present disclosure further includes:
If the judgment result is yes, judging whether the cluster resource utilization rate of the first cluster with the LB instance belonging to the first subnet reaches a preset cluster resource utilization rate threshold value or not based on the cluster information of each cluster in the target cloud platform;
if yes, deploying the target LB instance in a second cluster with the lowest cluster resource utilization rate in the rest clusters; the rest clusters refer to clusters except the first cluster in the target cloud platform.
In one embodiment, deploying the target LB instance on a second cluster with a lowest cluster resource utilization among the remaining clusters comprises:
Binding the virtual IP address of the target LB instance to a port of the second cluster; the port refers to a port which is created when the cluster applies for the assignment of the subnet to the software defined network SDN controller of the target cloud platform.
In an embodiment, after determining whether the cluster resource usage of the first cluster having the LB instance belonging to the first subnet reaches the preset cluster resource usage threshold, the method provided by the present disclosure further includes:
if the judgment result is negative, deploying the target LB instance in the first cluster.
In an embodiment, the method provided by the present disclosure further comprises:
responding to the third cluster with faults in the target cloud platform, and acquiring at least one first LB instance deployed on the third cluster and a subnet to which each first LB instance belongs based on cluster information of each cluster in the target cloud platform;
Judging whether an LB instance which is the same as the subnet to which the first LB instance belongs exists in the healthy cluster or not based on cluster information of each cluster in the target cloud platform; healthy clusters refer to clusters in the target cloud platform which do not have faults;
if yes, deploying the corresponding first LB instance in the first health cluster; the first health cluster refers to a health cluster in which there are the same LB instances as the subnet to which the first LB instance belongs.
In an embodiment, after determining whether the same LB instance exists in the healthy cluster as the subnet to which the first LB instance belongs, the method provided by the present disclosure further includes:
if the judgment result is negative, deploying the corresponding first LB instance in the second health cluster; the second healthy cluster refers to a cluster with the lowest cluster resource utilization rate among healthy clusters in which no LB instances exist in the same subnet as the first LB instance.
Embodiments of a second aspect of the present disclosure provide a load balancer deployment apparatus, the apparatus comprising:
the acquisition unit is used for acquiring cluster information of each cluster in the target cloud platform, wherein the cluster information at least comprises a load balancer LB instance deployed by the corresponding cluster, a sub-network to which the LB instance belongs and a cluster resource utilization rate;
The judging unit is used for responding to a request for creating a target LB instance belonging to the first subnet and judging whether the LB instance belonging to the first subnet exists in the cluster of the target cloud platform or not;
The deployment unit is used for deploying the target LB instance on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform if the judgment result is negative.
An embodiment of a third aspect of the present disclosure proposes an electronic device, including:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described in the embodiments of the first aspect of the present disclosure.
An embodiment of a fourth aspect of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the embodiment of the first aspect of the present disclosure.
A fifth aspect embodiment of the present disclosure proposes a computer program product comprising a computer program which, when executed by a processor, implements the method described in the first aspect embodiment of the present disclosure.
In summary, the present disclosure proposes a load balancer deployment method, which includes: acquiring cluster information of each cluster in a target cloud platform, wherein the cluster information at least comprises a load balancer LB instance, a subnet to which the LB instance belongs and a cluster resource utilization rate of which the corresponding cluster is deployed; responding to a request for creating a target LB instance belonging to a first subnet, and judging whether the LB instance belonging to the first subnet exists in a cluster of a target cloud platform; if the judgment result is negative, the target LB instance is deployed on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform.
According to the scheme provided by the disclosure, the health state of each cluster is searched through the Load balancing service agent (Load balancing-as-a-SERVICE AGENT, lbaas-agent) management node, and the LB examples belonging to the first subnet in the cluster are determined according to the cluster index, so that the LB examples in each cluster can be efficiently positioned and managed, and the availability of the cluster is improved. If the LB instances belonging to the same subnet do not exist in the cluster of the target cloud platform, the target LB instance of the first subnet is deployed in the cluster with the lowest resource utilization rate, so that the load balancing performance of the cluster is improved, the utilization of cluster resources is facilitated, and the resource waste is avoided.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.
FIG. 1 is a functional block diagram of a soft load balancing deployment architecture provided by an embodiment of the present disclosure;
fig. 2 is a flow chart of a load balancer deployment method provided by an application example of the present disclosure;
FIG. 3 is a schematic flow chart of deploying a target LB instance in a second cluster according to an embodiment of the disclosure;
FIG. 4 is a functional block diagram of a load balancer deployment architecture provided by an example application of the present disclosure;
FIG. 5 is a schematic diagram of a process for across-cluster deployment of LB instances provided by an application example of the present disclosure;
Fig. 6 is a flowchart of a load balancer deployment method provided by an application example of the present disclosure;
FIG. 7 is a flowchart of deploying a first LB instance in a first health cluster according to an embodiment of the disclosure;
fig. 8 is a flowchart of yet another load balancer deployment method provided by an application example of the present disclosure;
fig. 9 is a schematic structural diagram of a load balancer deployment apparatus according to an embodiment of the present disclosure;
fig. 10 is a schematic diagram of a hardware composition structure of an electronic device according to an embodiment of the disclosure.
Detailed Description
Embodiments of the present disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present disclosure and are not to be construed as limiting the present disclosure.
In the related art, the LBs under the same subnet are generally distributed in the same cluster, so that cluster resources under the same subnet are not used uniformly, and when the cluster resources of the LBs under the same subnet are full, the LBs can not be continuously created under the subnet, and the new instance needs to be created by switching the subnet, so that the cluster utilization rate under the same subnet is reduced.
In the related art, a common load balancer deployment scheme is as follows:
(1) As shown in fig. 1, a multi-cluster mode is adopted for deployment, and clusters support expansion and contraction capacity;
(2) Each cluster comprises a main server and a standby server, and is used for supporting high availability of the main server and the standby server;
(3) One LB example comprises a pair of active and standby Linux virtual servers (Linux Virtual Server, LVS), wherein LVS clusters are configured and managed by a high-availability mechanism through KEEPALIVED to realize four-layer flow forwarding; the LVS back end distributes and deploys Nginx to realize seven-layer flow forwarding. When a user issues and creates information such as an LB instance, a monitor and the like from the cloud platform, different LVS+NGINX Namespaces (NAMESPCE) can be automatically virtual out in each server, so that tenant isolation scenes are met;
(4) When a user creates a load balancer LB1 by using a subnet 1 (subnet), lbaas-agent detects that LB instances under the subnet do not exist in all clusters, applies for distributing resources such as a virtual local area network (Virtual Local Area Network, VLAN) to a software defined network (Software Defined Network, SDN) controller (different VLAN ranges distributed by different cluster plans), creates ports and the like, carries out hierarchical binding operation, and when the user accesses the LB, SDN virtual tunnel endpoint (Virtual Tunnel Endpoint, VTEP) equipment carries out encapsulation and decapsulation of a virtual extensible local area network (Virtual Extensible LAN, VXLAN) to realize intercommunication of the created LB instance resources under the subnet with back-end servers under the same subnet, and then issues configuration of the LB1 into one of the clusters such as the cluster 1. When the subnet 1 is used to create the LB instance later, the allocated VLAN, port and other resources are used to bind the LB2 virtual Internet protocol (Virtual Internet Protocol, VIP) address, LVS and NGINX Internet protocol (Internet Protocol, IP) address to the port, and the configuration of LB2 is issued to the cluster 1 to realize the flow intercommunication. Each VLAN corresponds to one subnet, so that LBs under the same subnet are distributed on two servers in the same cluster. When NAMESPCE or other resources in cluster 1 are full, the creation of LB instances cannot continue using subnet 1.
The following drawbacks exist in the above scheme:
The distribution of the LBs under the same subnet in the same cluster is easy to cause uneven cluster resource usage, the occupancy rate of some cluster resources reaches 100%, the utilization rate of part of clusters is extremely low, when the cluster resources of the LBs under the same subnet are full, the LBs can not be continuously created under the subnet, the subnet 2 needs to be switched to create new instances, and the new instances fall on the idle clusters, which is unacceptable to cloud platform users. In addition, if the resource usage of the cluster 1 is full, when the cluster 1 fails, all LB instances already configured in the cluster are limited in that LB instances of the same subnet cannot be created into a new cluster, so that LB instances in the failed cluster cannot be migrated into a normal cluster.
In order to solve at least one defect in the related art, the disclosure provides a load balancer deployment method, which searches the health status of each cluster through lbaas-agent management nodes and determines the LB instance belonging to the first subnet in the cluster according to the cluster index, so that the LB instance in each cluster can be efficiently located and managed, and the availability of the cluster is improved. If the LB instances belonging to the same subnet do not exist in the cluster of the target cloud platform, the target LB instance of the first subnet is deployed in the cluster with the lowest resource utilization rate, so that the load balancing performance of the cluster is improved, the utilization of cluster resources is facilitated, and the resource waste is avoided.
The present disclosure will be described in further detail with reference to the accompanying drawings and specific embodiments.
As shown in fig. 2, a load balancer deployment method provided by an embodiment of the present disclosure includes the following steps:
step 201, obtaining cluster information of each cluster in the target cloud platform.
In one embodiment, cluster information for each cluster may be obtained by lbaas-agent management nodes.
In an embodiment, the cluster information at least includes the LB instance where the corresponding cluster has been deployed, the subnet to which the LB instance belongs, and the cluster resource usage.
In an embodiment, lbaas-agent management nodes can detect health status of different clusters periodically by monitoring agent heartbeats and store cluster health status in a database.
In one embodiment, lbaas-agent management nodes may also be used to monitor LB instance distribution.
In an embodiment, the database storing the health status of the cluster may be any one of MySQL, postgreSQL, neutron databases, which is not limited in this disclosure.
In an embodiment, by acquiring the cluster information of each cluster in the target cloud platform, the LB instance in each cluster can be efficiently located and managed, and the availability of the cluster is improved.
Step 202, in response to a request for creating a target LB instance belonging to a first subnet, determining whether there is a LB instance belonging to the first subnet in a cluster of the target cloud platform;
In one embodiment, the target LB instance refers to an LB instance created in response to a request to create a target LB instance belonging to the first subnet.
In an embodiment, the first subnetwork may be a logical subnetwork divided in a local area network or a wide area network.
In an embodiment, the LB instance of the first subnet may be one or more, which is not limited in this disclosure.
In an embodiment, when the lbaas-agent management node receives a request for creating a target LB instance belonging to the first subnet, the lbaas-agent management node may query a database, where the database includes a plurality of LB instance IDs and corresponding subnet IDs, and may search for LB instances under the first subnet according to the first subnet ID and the corresponding LB instance IDs in the database.
Step 203, if the result of the determination is no, deploying the target LB instance on the cluster with the lowest cluster resource utilization based on the cluster information of each cluster in the target cloud platform.
In an embodiment, a database is queried through lbaas-agent management and control nodes, and if an LB instance under a first subnet cannot be found according to a first subnet ID and a corresponding LB instance ID, a target LB instance is deployed on a cluster with the lowest cluster resource utilization rate based on cluster information of each cluster in a target cloud platform.
In an embodiment, if the LB instance under the first subnet cannot be found according to the first subnet ID and the corresponding LB instance ID, the step of deploying the target LB instance in the cluster with the lowest cluster resource utilization based on the cluster information of each cluster in the target cloud platform includes:
acquiring cluster information of each cluster in a target cloud platform through lbaas-agent management and control nodes, and determining the cluster with the lowest cluster resource utilization rate;
And on the selected cluster with the lowest cluster resource utilization rate, deploying the target LB instance to the cluster with the lowest cluster resource utilization rate by using a deployment tool provided by the target cloud platform, such as Docker, kubernetes, ansible, chef, puppet.
In an embodiment, the target LB instance is deployed on the cluster with the lowest cluster resource utilization rate, so that the load balancing performance of the cluster is improved, the utilization of cluster resources is facilitated, and the resource waste is avoided.
In summary, the present disclosure proposes a load balancer deployment method, which includes: acquiring cluster information of each cluster in a target cloud platform, wherein the cluster information at least comprises a load balancer LB instance, a subnet to which the LB instance belongs and a cluster resource utilization rate of which the corresponding cluster is deployed; responding to a request for creating a target LB instance belonging to a first subnet, and judging whether the LB instance belonging to the first subnet exists in a cluster of a target cloud platform; if the judgment result is negative, the target LB instance is deployed on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform.
According to the scheme provided by the disclosure, through lbaas-agent management and control nodes, the health state of each cluster is searched, and the LB examples belonging to the first subnet in the cluster are determined according to the cluster index, so that the LB examples in each cluster can be efficiently positioned and managed, and the availability of the cluster is improved. If the LB instances belonging to the same subnet do not exist in the cluster of the target cloud platform, the target LB instance of the first subnet is deployed in the cluster with the lowest resource utilization rate, so that the load balancing performance of the cluster is improved, the utilization of cluster resources is facilitated, and the resource waste is avoided.
In one embodiment, as shown in fig. 3, following step 202, the method provided by the present disclosure includes the steps of:
Step 301, if the result is yes, based on the cluster information of each cluster in the target cloud platform, judging whether the cluster resource utilization rate of the first cluster with the LB instance belonging to the first subnet reaches a preset cluster resource utilization rate threshold;
In an embodiment, the first cluster refers to a cluster under the first subnet.
In one embodiment, the preset cluster resource usage threshold refers to a threshold value that is full of cluster resource usage.
In one embodiment, the preset cluster resource usage threshold may be used to measure whether an LB instance of the first cluster may be created under the first subnet.
In an embodiment, since the LB instances of the first subnet are generally distributed in the same cluster, each cluster resource in the first subnet is used unevenly, which reduces the cluster utilization of the first subnet. Therefore, whether the cluster resource utilization rate of the first cluster of the LB instance of the first subnet reaches a preset cluster resource utilization rate threshold is used as a judgment standard for measuring whether the LB instance of the first cluster can be created under the first subnet, and the utilization rate of the cluster under the first subnet is improved.
Step 302, if the result is yes, the target LB instance is deployed in a second cluster with the lowest cluster resource utilization rate in the remaining clusters.
In an embodiment, the second cluster refers to another cluster under the first subnet than the first cluster.
In an embodiment, the remaining clusters refer to clusters in the target cloud platform other than the first cluster.
In one embodiment, deploying the target LB instance on a second cluster with a lowest cluster resource utilization among the remaining clusters comprises:
Assigning a Virtual Local Area Network (VLAN) to the second cluster, creating a port (self_port), creating a gateway port (gateway_port) from self_port and VLAN;
binding the virtual IP address of the target LB instance to a second cluster port;
Creating ports of LVS and NGINX, binding addresses of LVS and NGINX into a second cluster self_port, creating KEEPALIVED, NGINX configuration file, and forwarding traffic to the second cluster when accessing LB instance VIP (virtual IP address).
In an embodiment, since the LB instance belonging to the first subnet exists in the clusters of the target cloud platform, and the cluster resource usage of the first cluster of the LB instance belonging to the first subnet reaches the preset cluster resource usage threshold. Therefore, the cluster utilization rate in the first subnet is improved by deploying the target LB instance in the second cluster with the lowest cluster resource utilization rate in the remaining clusters.
In one embodiment, step 302 includes:
binding the virtual IP address of the target LB instance to a port of the second cluster.
In an embodiment, the port refers to a port created when the cluster applies for assignment of the subnet to the SDN controller of the target cloud platform.
In one embodiment, binding the virtual IP address of the target LB instance to the port of the second cluster comprises:
Determining a network topology of the second cluster: the network structure of the second cluster comprises network equipment, a subnet, an IP address range and the like;
Configuring a virtual IP address: the configuration of the virtual IP address of the target LB instance can be completed through a management console and a command line interface of the target cloud platform;
Configuring port forwarding rules: configuring port forwarding rules to forward traffic from the virtual IP address to the ports of the second cluster;
Configuring network devices of the second cluster: ensuring that network devices in the second cluster can accept traffic from the virtual IP address;
Testing and verifying: after the configuration is completed, testing and verification are performed to ensure that the virtual IP address is bound to the port of the second cluster. Verification may be performed by sending traffic and checking whether ports in the second cluster are receiving traffic correctly;
In an embodiment, the virtual IP address of the target LB instance is bound to the port of the second cluster, so that high availability of the second cluster can be achieved, and if a certain node in the cluster fails and is not available, the target LB instance can automatically forward the traffic to a healthy node, so that normal operation of the service is ensured.
In an embodiment, after step 301, the method provided by the present disclosure further includes:
if the judgment result is negative, deploying the target LB instance in the first cluster.
In an embodiment, if the LB instance under the first subnet can be found according to the first subnet ID and the corresponding LB instance ID in the database, and the cluster usage of the first cluster does not reach the preset cluster resource usage threshold, the target LB instance is deployed in the first cluster.
In an embodiment, if the LB instance under the first subnet can be found according to the first subnet ID and the corresponding LB instance ID in the database, when the target LB instance is deployed in the first cluster, the target LB instance does not need to apply for VLAN assignment and port creation to the SDN controller, and only needs to bind the virtual IP address of the target LB instance to the port of the first cluster, thereby improving the configuration efficiency of the target LB instance.
As shown in fig. 4 and 5: fig. 4 is a schematic block diagram of a load balancer deployment architecture provided by the application example of the present disclosure, and fig. 5 is a process schematic diagram of LB instance cross-cluster deployment provided by the application example of the present disclosure. Taking an example that an LB instance belonging to a first subnet exists in a first cluster of a target cloud platform, and the cluster utilization rate of the first cluster reaches a preset cluster resource utilization rate threshold, continuing to create the LB instance under the first subnet, the method comprises the following steps:
lbaas-after receiving the request for creating the LB instance, the agent management node queries the database and selects health status as the active cluster;
When the LB instances under the first subnet already exist on the first cluster and the resource utilization rate (usage_rate) is 1 and the number of LB instances under the first subnet on other clusters is 0, the other clusters simultaneously apply for distributing a Virtual Local Area Network (VLAN), a creation port (self_port) and a gateway port (gateway_port) to the SDN controller in parallel except the first cluster;
The second cluster allocates VLAN200, creates a second cluster self_port, creates gateway_port according to the second cluster self_port and VLAN200, the third cluster allocates VLAN300, creates a third cluster self_port, creates gateway_port according to the third cluster self_port and VLAN300, and synchronously creates corresponding resources for the rest of the other clusters; when an LB instance is configured to a certain cluster, only the VIP address of the LB instance is bound to self_port, and meanwhile, configuration files of keepalive and nginx are created, so that traffic can be forwarded to the cluster normally;
Querying a database to select a cluster with the least cluster resource utilization rate, at the moment, confirming a cluster configured by an LB instance, such as a second cluster, binding the address of the LB instance VIP to a self_port created by the second cluster, creating ports of LVS and NGINX, binding the addresses of LVS and NGINX to the second cluster self_port, creating a KEEPALIVED, NGINX configuration file, and forwarding the flow to the second cluster when accessing the LB instance VIP;
lbaas-agent management and control node periodically detects the number of LB instances created on all clusters, CPU utilization rate, memory utilization rate, disk utilization rate, network card access flow information, and calculates and updates the resource utilization rate, subnet ID, cluster ID and LB instance number information of each cluster in the database.
As shown in fig. 6, fig. 6 is a flowchart of a load balancer deployment method provided by an application example of the present disclosure, including the following steps:
Step 601, obtaining cluster information of each cluster in a target cloud platform;
Step 602, in response to a request for creating a target LB instance belonging to the first subnet, determining whether there is a LB instance belonging to the first subnet in the cluster of the target cloud platform;
If yes, go to step 603;
if not, go to step 604;
step 603, based on the cluster information of each cluster in the target cloud platform, judging whether the cluster resource utilization rate of the first cluster with the LB instance belonging to the first subnet reaches a preset cluster resource utilization rate threshold;
if yes, go to step 604;
if not, go to step 605;
Step 604, deploying the target LB instance in a second cluster with the lowest cluster resource utilization rate in the remaining clusters;
in step 605, a target LB instance is deployed on a first cluster.
In an embodiment, as shown in fig. 7, the method provided by the present disclosure further includes the following steps:
Step 701, responding to a third cluster with faults in a target cloud platform, and acquiring at least one first LB instance deployed on the third cluster and a subnet to which each first LB instance belongs based on cluster information of each cluster in the target cloud platform;
In an embodiment, searching a third cluster ID in the database through lbaas-agent management and control nodes, searching a subnet ID corresponding to at least one first LB instance ID under the third cluster according to the third cluster ID in the database, and finding a first LB instance according to the subnet ID corresponding to the first LB instance ID.
In one embodiment, the configuration information such as listener, pool, health monitor, etc. is found according to the first LB instance ID. Wherein listener is responsible for receiving a client request and forwarding the request to a node in the cluster, when listener receives the request, pool is used for processing the request, ensuring high availability and load balancing of the cluster, health monitor is used for monitoring health status of the node in the cluster, and if a node failure is monitored, LB instance stops forwarding traffic to the node.
In an embodiment, by acquiring at least one first LB instance deployed on the third cluster and a subnet to which each first LB instance belongs, network isolation conditions and LB instance configuration information between different LB instances are known, and migration processing time for migrating the LB instance from the failed cluster to the healthy cluster is optimized.
Step 702, based on the cluster information of each cluster in the target cloud platform, determining whether there is an LB instance in the healthy cluster, which is the same as the subnet to which the first LB instance belongs.
In one embodiment, a healthy cluster refers to a cluster in a target cloud platform that has not failed.
Step 703, if the determination result is yes, deploying the corresponding first LB instance in the first health cluster.
In one embodiment, the first health cluster refers to a health cluster in which there are the same LB instances as the subnet to which the first LB instance belongs.
In an embodiment, if the same LB instance exists in the healthy cluster as the subnet to which the first LB instance belongs, only the VIP address of the first LB instance needs to be updated to the port of the first healthy cluster, and meanwhile, the configuration information of the LB instance is deployed in the first cluster, so that VLAN application and port resource creation are not required to be applied to the SDN controller again, and the fault processing duration is optimized.
In one embodiment, the configuration information of the LB instance includes listener, pool, health monitor, and other configuration information.
In an embodiment, following step 702, the method provided by the present disclosure further comprises:
if the judgment result is negative, the corresponding first LB instance is deployed in the second health cluster.
In an embodiment, the second healthy cluster refers to a cluster with the lowest cluster resource usage among healthy clusters that do not have the same LB instance as the subnet to which the first LB instance belongs.
In an embodiment, if the same LB instance as the subnet to which the first LB instance belongs does not exist in the healthy cluster, the number of LB instances of different subnets in the failed cluster is from high to low, and the LB instances are deployed into the cluster with the low-to-high cluster resource utilization rate respectively, and the cluster port information is updated and the LB instance configuration information is redeployed.
In an embodiment, the corresponding first LB instance is deployed in the second healthy cluster, so that load balancing performance of the cluster is improved, utilization of cluster resources is facilitated, and resource waste is avoided.
As shown in fig. 8, fig. 8 is a flowchart of a load balancer deployment method provided by an application example of the present disclosure, including the following steps:
Step 801, in response to a third cluster with a fault in the target cloud platform, acquiring at least one first LB instance deployed on the third cluster and a subnet to which each first LB instance belongs based on cluster information of each cluster in the target cloud platform;
Step 802, judging whether an LB instance which is the same as a subnet to which a first LB instance belongs exists in a healthy cluster or not based on cluster information of each cluster in a target cloud platform;
if yes, go to step 803;
If not, go to step 804;
step 803, deploying the corresponding first LB instance in a first health cluster;
step 804, deploy the corresponding first LB instance in the second health cluster.
In order to implement the load balancer deployment method provided by the embodiment of the present disclosure, the embodiment of the present disclosure further provides a load balancer deployment device, as shown in fig. 9. Fig. 9 is a schematic structural diagram of a load balancer deployment apparatus according to an embodiment of the present disclosure, where the load balancer deployment apparatus 900 includes:
The acquiring unit 901 is configured to acquire cluster information of each cluster in the target cloud platform, where the cluster information at least includes a load balancer LB instance where the corresponding cluster has been deployed, a subnet to which the LB instance belongs, and a cluster resource utilization rate;
A judging unit 902, configured to judge whether an LB instance belonging to the first subnet exists in a cluster of the target cloud platform in response to a request for creating the target LB instance belonging to the first subnet;
the deployment unit 903 is configured to deploy the target LB instance to a cluster with the lowest cluster resource utilization based on the cluster information of each cluster in the target cloud platform if the determination result is negative.
In one embodiment, the deployment unit 903 is specifically configured to:
If the judgment result is yes, judging whether the cluster resource utilization rate of the first cluster with the LB instance belonging to the first subnet reaches a preset cluster resource utilization rate threshold value or not based on the cluster information of each cluster in the target cloud platform;
if yes, deploying the target LB instance in a second cluster with the lowest cluster resource utilization rate in the rest clusters; the rest clusters refer to clusters except the first cluster in the target cloud platform.
In an embodiment, the load balancer deployment apparatus 900 further comprises a binding unit for:
Binding the virtual IP address of the target LB instance to a port of the second cluster; the port refers to a port which is created when the cluster applies for the assignment of the subnet to the software defined network SDN controller of the target cloud platform.
In one embodiment, the deployment unit 903 is specifically configured to:
if the judgment result is negative, deploying the target LB instance in the first cluster.
In an embodiment, load balancer deployment apparatus 900 further includes a first LB instance unit configured to deploy:
responding to the third cluster with faults in the target cloud platform, and acquiring at least one first LB instance deployed on the third cluster and a subnet to which each first LB instance belongs based on cluster information of each cluster in the target cloud platform;
Judging whether an LB instance which is the same as the subnet to which the first LB instance belongs exists in the healthy cluster or not based on cluster information of each cluster in the target cloud platform; healthy clusters refer to clusters in the target cloud platform which do not have faults;
if yes, deploying the corresponding first LB instance in the first health cluster; the first health cluster refers to a health cluster in which there are the same LB instances as the subnet to which the first LB instance belongs.
In one embodiment, a first LB instance unit is deployed, specifically for:
if the judgment result is negative, deploying the corresponding first LB instance in the second health cluster; the second healthy cluster refers to a cluster with the lowest cluster resource utilization rate among healthy clusters in which no LB instances exist in the same subnet as the first LB instance.
It should be noted that: in the load balancer deployment apparatus provided in the above embodiment, only the division of each program module is used for illustration, and in practical application, the above processing allocation may be performed by different program modules according to needs, that is, the internal structure of the load balancer deployment apparatus is divided into different program modules, so as to complete all or part of the above processing. In addition, the load balancer deployment apparatus provided in the above embodiment and the load balancer deployment method embodiment provided in the embodiments of the present disclosure belong to the same concept, and detailed implementation processes of the load balancer deployment apparatus are referred to the method embodiment, and are not repeated here.
Fig. 10 is a schematic diagram of a hardware composition structure of an electronic device according to an embodiment of the disclosure, and as shown in fig. 10, the electronic device 1000 includes at least one processor 1002; and a memory 1001 communicatively coupled to the at least one processor 1002; the memory 1001 stores instructions executable by the at least one processor 1002, and the instructions are executed by the at least one processor 1002 to implement the steps of the load balancer deployment method of the embodiment of the present disclosure.
Optionally, the electronic device may be specifically a load balancer deployment apparatus according to the embodiment of the present application, and the electronic device may implement a corresponding flow implemented by the load balancer deployment apparatus in each method according to the embodiment of the present application, which is not described herein for brevity.
It will be appreciated that a communication interface 1003 is also included in the electronic device. The various components in the electronic device are coupled together by a bus system 1004. It is to be appreciated that the bus system 1004 serves to facilitate connective communication between these components. The bus system 1004 includes a power bus, a control bus, and a status signal bus in addition to the data bus. The various buses are labeled in fig. 10 as bus system 1004 for clarity of illustration.
It will be appreciated that the memory 1001 may be either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. The non-volatile Memory may be, among other things, a Read Only Memory (ROM), a programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read-Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable Read-Only Memory (EEPROM, ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), Magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk-Only (CD-ROM, compact Disc Read-Only Memory); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be random access memory (RAM, random Access Memory) which acts as external cache memory. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronous Dynamic Random Access Memory), and, Double data rate synchronous dynamic random access memory (DDRSDRAM, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, syncLink Dynamic Random Access Memory), Direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory 1001 described in embodiments of the present invention is intended to comprise, without being limited to, these and any other suitable types of memory.
The methods disclosed in the embodiments of the present disclosure described above may be applied to the processor 1002 or implemented by the processor 1002. The processor 1002 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the methods described above may be performed by integrated logic circuitry in hardware or instructions in software in the processor 1002. The processor 1002 may be a general purpose processor, DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 1002 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiment of the invention can be directly embodied in the hardware of the decoding processor or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium in the memory 1001 and the processor 1002 reads information in the memory 1001 to perform the steps of the method as described above in connection with the hardware thereof.
In an exemplary embodiment, the electronic device may be implemented by one or more Application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs, programmable Logic Device), complex programmable logic devices (CPLDs, complex Programmable Logic Device), FPGAs, general purpose processors, controllers, MCUs, microprocessors, or other electronic elements for performing the aforementioned methods.
The disclosed embodiments also provide a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the steps of implementing the load balancer deployment method of the embodiments of the present invention.
Optionally, the computer readable storage medium may be applied to the load balancer deployment apparatus in the embodiment of the present application, and the computer instructions cause a computer to execute corresponding processes implemented by the load balancer deployment apparatus in the methods in the embodiments of the present application, which are not described herein for brevity.
The disclosed embodiments also provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of the load balancer deployment method provided by the embodiments of the present invention.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of units is only one logical function division, and there may be other divisions in actual implementation, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described as separate units may or may not be physically separate, and units displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
Or the above-described integrated units of the invention may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
The foregoing is merely illustrative embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the technical scope of the present invention, and the invention should be covered. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims (10)

1. A method for deploying a load balancer, comprising:
Acquiring cluster information of each cluster in a target cloud platform, wherein the cluster information at least comprises a corresponding load balancer LB instance deployed by the cluster, a subnet to which the LB instance belongs and a cluster resource utilization rate;
Responding to a request for creating a target LB instance belonging to a first subnet, and judging whether the LB instance belonging to the first subnet exists in the cluster of the target cloud platform;
If the judgment result is negative, the target LB instance is deployed on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform.
2. The method of claim 1, wherein after said determining whether there is an LB instance belonging to the first subnet in the cluster of the target cloud platform, the method further comprises:
If so, judging whether the cluster resource utilization rate of a first cluster of the LB instance belonging to the first subnet reaches a preset cluster resource utilization rate threshold or not based on cluster information of each cluster in the target cloud platform;
If yes, deploying the target LB instance in a second cluster with the lowest cluster resource utilization rate in the rest clusters; the remaining clusters refer to clusters except the first cluster in the target cloud platform.
3. The method of claim 2, wherein the deploying the target LB instance in the second cluster with the lowest cluster resource utilization among the remaining clusters comprises:
Binding the virtual IP address of the target LB instance to a port of the second cluster; the port refers to a port which is created when the cluster applies for the assignment of a subnet to a software defined network SDN controller of the target cloud platform.
4. The method of claim 2, wherein after determining whether the cluster resource usage of the first cluster in which the LB instance belongs to the first subnet reaches a preset cluster resource usage threshold, the method further comprises:
if not, deploying the target LB instance in the first cluster.
5. The method of claim 1, wherein the method further comprises:
Responding to the existence of a third cluster with faults in the target cloud platform, and acquiring at least one first LB instance deployed on the third cluster and a subnet to which each first LB instance belongs based on cluster information of each cluster in the target cloud platform;
Judging whether LB instances which are the same as the subnet to which the first LB instance belongs exist in a healthy cluster or not based on cluster information of each cluster in the target cloud platform; the healthy clusters refer to clusters which do not have faults in the target cloud platform;
If yes, deploying the corresponding first LB instance in the first health cluster; the first health cluster refers to a health cluster in which there are the same LB instances as the subnet to which the first LB instance belongs.
6. The method of claim 5, wherein after determining whether the same LB instance exists in the healthy cluster as the subnet to which the first LB instance belongs, the method further comprises:
if the judgment result is negative, deploying the corresponding first LB instance in the second health cluster; the second healthy cluster refers to a healthy cluster in which the same LB instance as the subnet to which the first LB instance belongs does not exist, and the cluster resource utilization rate is the lowest.
7. A load balancer deployment apparatus, comprising:
The system comprises an acquisition unit, a target cloud platform and a storage unit, wherein the acquisition unit is used for acquiring cluster information of each cluster in the target cloud platform, and the cluster information at least comprises a corresponding load balancer LB instance in which the cluster is deployed, a subnet to which the LB instance belongs and a cluster resource utilization rate;
The judging unit is used for responding to a request for creating a target LB instance belonging to a first subnet and judging whether the LB instance belonging to the first subnet exists in the cluster of the target cloud platform or not;
And the deployment unit is used for deploying the target LB instance on the cluster with the lowest cluster resource utilization rate based on the cluster information of each cluster in the target cloud platform if the judgment result is negative.
8. An electronic device, comprising:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.
9. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 6.
CN202410581669.8A 2024-05-11 2024-05-11 Load balancer deployment method, device, electronic equipment, storage medium and product Active CN118158222B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410581669.8A CN118158222B (en) 2024-05-11 2024-05-11 Load balancer deployment method, device, electronic equipment, storage medium and product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410581669.8A CN118158222B (en) 2024-05-11 2024-05-11 Load balancer deployment method, device, electronic equipment, storage medium and product

Publications (2)

Publication Number Publication Date
CN118158222A true CN118158222A (en) 2024-06-07
CN118158222B CN118158222B (en) 2024-08-09

Family

ID=91289051

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410581669.8A Active CN118158222B (en) 2024-05-11 2024-05-11 Load balancer deployment method, device, electronic equipment, storage medium and product

Country Status (1)

Country Link
CN (1) CN118158222B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101447939A (en) * 2008-12-16 2009-06-03 中国移动通信集团北京有限公司 Functional distribution method and load balancer
CN107733991A (en) * 2017-09-19 2018-02-23 大唐网络有限公司 One kind provides load balancing service implementation method based on OpenStack frameworks
CN111666131A (en) * 2020-07-03 2020-09-15 腾讯科技（深圳）有限公司 Load balancing distribution method and device, computer equipment and storage medium
CN115604272A (en) * 2022-11-30 2023-01-13 苏州浪潮智能科技有限公司（Cn） Load balancing method, device, system creating method, device and medium
CN115729671A (en) * 2022-11-16 2023-03-03 浙江大华技术股份有限公司 Resource scheduling method and related device
CN116846900A (en) * 2022-11-30 2023-10-03 天翼数字生活科技有限公司 Load balancing intelligent scheduling method, device and equipment
CN117221326A (en) * 2023-08-28 2023-12-12 中国移动(浙江)创新研究院有限公司 Software load balancing scheduling method, system, electronic equipment and storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101447939A (en) * 2008-12-16 2009-06-03 中国移动通信集团北京有限公司 Functional distribution method and load balancer
CN107733991A (en) * 2017-09-19 2018-02-23 大唐网络有限公司 One kind provides load balancing service implementation method based on OpenStack frameworks
CN111666131A (en) * 2020-07-03 2020-09-15 腾讯科技（深圳）有限公司 Load balancing distribution method and device, computer equipment and storage medium
CN115729671A (en) * 2022-11-16 2023-03-03 浙江大华技术股份有限公司 Resource scheduling method and related device
CN115604272A (en) * 2022-11-30 2023-01-13 苏州浪潮智能科技有限公司（Cn） Load balancing method, device, system creating method, device and medium
CN116846900A (en) * 2022-11-30 2023-10-03 天翼数字生活科技有限公司 Load balancing intelligent scheduling method, device and equipment
CN117221326A (en) * 2023-08-28 2023-12-12 中国移动(浙江)创新研究院有限公司 Software load balancing scheduling method, system, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN118158222B (en) 2024-08-09

Similar Documents

Publication Publication Date Title
CN110113441B (en) Computer equipment, system and method for realizing load balance
US10846185B2 (en) Method for processing acquire lock request and server
US9952947B2 (en) Method and system for processing fault of lock server in distributed system
CN110730250B (en) Information processing method and device, service system and storage medium
CN112165502B (en) Service discovery system, method and second server
US11036535B2 (en) Data storage method and apparatus
EP3806395B1 (en) Virtual network function (vnf) deployment method and apparatus
US11153173B1 (en) Dynamically updating compute node location information in a distributed computing environment
CN107666493B (en) Database configuration method and equipment thereof
CN110474802B (en) Equipment switching method and device and service system
US11349706B2 (en) Two-channel-based high-availability
CN108933844B (en) Method and equipment for providing DHCP service
CN106452836B (en) main node setting method and device
US11917001B2 (en) Efficient virtual IP address management for service clusters
CN112202853B (en) Data synchronization method, system, computer device and storage medium
US20240211435A1 (en) Information processing method and apparatus
CN111770159B (en) Service discovery system, method, first server and storage medium
CN118158222B (en) Load balancer deployment method, device, electronic equipment, storage medium and product
CN114157663A (en) Cloud data access method and cloud server
CN115687250A (en) Storage method, equipment, system and computer storage medium
CN110110004B (en) Data operation method, device and storage medium
CN111382326A (en) Instance group creation method, instance group creation device, instance group creation server and storage medium
WO2023273483A1 (en) Data processing system and method, and switch
CN117714476A (en) Cloud coil control method, cloud coil control system, electronic equipment and storage medium
WO2023246241A1 (en) Data processing system, data processing method and apparatus, and related device

Legal Events

Date Code Title Description
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant