WO2024060596A1 - 面向多K8s集群的Spark协同作业方法及装置 - Google Patents

面向多K8s集群的Spark协同作业方法及装置 Download PDF

Info

Publication number
WO2024060596A1
WO2024060596A1 PCT/CN2023/088148 CN2023088148W WO2024060596A1 WO 2024060596 A1 WO2024060596 A1 WO 2024060596A1 CN 2023088148 W CN2023088148 W CN 2023088148W WO 2024060596 A1 WO2024060596 A1 WO 2024060596A1
Authority
WO
WIPO (PCT)
Prior art keywords
cluster
slave
unit
executor
spark
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/088148
Other languages
English (en)
French (fr)
Inventor
高丰
白文媛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Lab
Original Assignee
Zhejiang Lab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Lab filed Critical Zhejiang Lab
Priority to US18/554,450 priority Critical patent/US11954525B1/en
Publication of WO2024060596A1 publication Critical patent/WO2024060596A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/2895—Intermediate processing functionally located close to the data provider application, e.g. reverse proxies
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present disclosure relates to the technical fields of cloud computing and big data, and in particular to Spark collaborative computing, operation methods and devices for multiple K8s clusters.
  • Apache Spark is a fast and general computing engine designed for large-scale data processing.
  • Spark was born in the AMP Laboratory (the Algorithms, Machines and People lab) of the University of California, Berkeley, and was open sourced in 2010.
  • Spark was donated to the Apache Software Foundation and became an Apache top-level project in 2014.
  • Today, Spark has become one of the commonly used tools for small and large enterprises and research institutions, and the mature applications accumulated by the Spark platform have become an important asset in the data industry.
  • K8s (Kubernetes) is an open source container scheduling and orchestration platform. Because Kubernetes applicable service abstraction supports naming and load balancing, and organizes multi-container scheduling unit Pod (Plain Old Documentation) through labels, it achieves better flexibility, availability and load balancing. sex. Therefore, in the wave of cloud native (Cloud Native) technology development, Kubernetes has become the operating system of the current cloud era along with the development of container technology. As the de facto standard in the field of container orchestration and a key project in the field of cloud native, Kubernetes has become the core technology that engineers in the cloud native era most need to understand and practice.
  • cloud native Cloud Native
  • the present disclosure adopts the following technical solution.
  • the present disclosure provides a multi-cluster oriented collaborative operation method, which is applied to a collaboration center and includes the following steps.
  • Step S101 Specify the primary cluster and the secondary cluster based on the job status information submitted by the user.
  • a job refers to a collection of work that the user requires the computing system to do during a solution or transaction processing process. It includes user programs, required data set control commands, etc.
  • a job is composed of a series of ordered steps. Executing a job will involve computing, storage, etc. on multiple network nodes; job status information includes the policy selected by the user. strategy and computing resources, data storage and other status information.
  • Step S102 Send the job description file to the master cluster so that the master cluster can establish the driver unit; submit the job description file to the slave cluster so that the slave cluster can create the slave cluster executor unit.
  • Step S103 Receive the registration information of the driver unit sent by the master cluster to complete the registration of the driver unit in the collaboration center, so that the slave cluster sends the registration information to the registered driver unit to complete the registration of the slave cluster executor unit in the driver unit, so that The registered slave cluster executor unit executes the jobs sent by the master cluster;
  • the collaboration center While receiving the registration information of the driver unit sent by the main cluster, the collaboration center also sends a reverse proxy startup request to the main cluster, and uses the feedback information established by the slave cluster executor unit to enable the driver unit to determine the establishment information of the slave cluster executor unit. , activate the reverse proxy unit, so that the slave cluster sends the address and credential information of the slave cluster executor unit to connect with the reverse proxy unit of the master cluster. If the connection is successful, the registration information is sent to the driver unit to complete the slave cluster. Registration of the actuator unit in the drive unit.
  • Step S104 After the execution of the job is completed, release the resources occupied by the execution of the job.
  • step S101 the collaboration center determines whether it is a single cluster job based on the job status information submitted by the user. If so, a single cluster is selected to execute the business and the occupied resources are released after the execution is completed. Otherwise, a master cluster and a slave cluster are specified and step S102 is executed.
  • step S102 the job description file is sent to the main cluster, so that after the main cluster starts the job, the main cluster executor unit is also established; in step S103, the registration information of the driver unit sent by the main cluster is received to After completing the registration of the driver unit in the collaboration center, the main cluster executor unit also sends the registration information of the main cluster executor unit to the registered driver unit to complete the registration of the main cluster executor unit in the driver unit, so that the registered main cluster Executor units execute jobs sent by the master cluster.
  • step S103 while the slave cluster sends registration information to the registered driver unit, it also sends a heartbeat message to the master cluster, so that the master cluster determines whether the connection is successful based on the registration information and heartbeat message of the slave cluster. , if the master cluster receives the registration information and the heartbeat message of the group of slave cluster executor units, the connection is successful and the job is sent; otherwise, the connection fails and the wait continues.
  • the present disclosure provides a multi-cluster collaborative operation device, including a job interface module, a job assignment module and a resource interface module.
  • the job assignment module includes a cluster specifying module, a job description file issuing module, a registration information issuing module and a resource releasing module.
  • the job interface module is used to obtain jobs submitted by users.
  • the cluster designation module is used to designate the master cluster and slave cluster based on the job status information submitted by the user.
  • the job description file delivery module is used to send the job description file to the master cluster so that the master cluster can establish the driver unit; and submit the job description file to the slave cluster so that the slave cluster can establish the slave cluster executor unit.
  • the registration information delivery module is used to send driver unit registration information to the main cluster to complete the registration of the driver unit in the collaboration center, and to enable the slave cluster to send registration information to the registered driver unit to complete the registration of the slave cluster executor unit in the driver unit. Register to enable the registered slave cluster executor unit to execute jobs sent by the master cluster.
  • the collaboration center While receiving the registration information of the driver unit sent by the main cluster, the collaboration center also sends a reverse proxy start request to the main cluster, and uses the feedback information established by the group of slave cluster executor units to enable the driver unit to determine the slave cluster executor unit.
  • the establishment information of the reverse proxy unit is activated, so that the slave cluster sends the address and credential information of the slave cluster executor unit to the master cluster to connect with the reverse proxy unit of the master cluster. If the slave cluster communicates with the master cluster's reverse proxy unit, If the connection to the agent unit is successful, the slave cluster sends the registration information of the group of slave cluster executor units to the driver unit to complete the registration of the group of slave cluster executor units in the driver unit.
  • the resource release module is used to release occupied resources after job execution is completed.
  • the resource interface module is used to obtain the resource status of each cluster.
  • the present disclosure provides a multi-cluster oriented collaborative operation method, which is applied to the main cluster and includes the following steps.
  • Step S201 Obtain the job description file sent by the collaboration center and create a driver unit.
  • Step S202 Send the driver unit registration information to the collaboration center, complete the registration of the driver unit in the collaboration center, obtain the registration information sent from the cluster executor unit through the registered driver unit, and complete the registration of the slave cluster executor unit in the driver unit.
  • the slave cluster executor is created from the slave cluster based on the job description file of the collaboration center.
  • the main cluster While the main cluster sends the registration information of the driver unit to the collaboration center, it also obtains a reverse proxy startup request, and uses the feedback information established by the group of slave cluster executor units to enable the driver unit to determine the group of slave cluster executor units.
  • the establishment information of the cluster executor unit activates the reverse proxy unit, so that the slave cluster sends the address and credential information of the group of slave cluster executor units to connect with the reverse proxy unit of the master cluster. If the connection is successful, then The slave cluster sends the registration information of the group of slave cluster executor units to the driver unit to complete the registration of the group of slave cluster executor units in the driver unit.
  • Step S203 Send the job to the registered slave cluster, so that the registered slave cluster executor unit executes the job.
  • the present disclosure provides a multi-cluster oriented collaborative operation method, which is applied to slave clusters and includes the following steps.
  • Step S301 Obtain the job description file sent by the collaboration center and establish the slave cluster executor unit;
  • Step S302 Send registration information to the registered driver unit to complete the registration of the slave cluster executor unit in the driver unit; the driver unit is established by the master cluster according to the job description file of the collaboration center; the registration of the driver unit is based on the job description file received by the collaboration center Drive unit registration information sent by the main cluster, registration completed in the collaboration center.
  • Step S303 Obtain the job sent by the main cluster and execute the job through the slave cluster executor unit.
  • This disclosure provides a Spark collaborative computing method for multiple K8s clusters, which is applied to a collaboration center and includes the following steps.
  • Step S401 Specify the K8s master cluster and the K8s slave cluster based on the job status information of the computing engine Spark submitted by the user.
  • Step S402 Send the Yaml file of the job description to the K8s API interface of the K8s main cluster, so that after the K8s main cluster receives the submission request of the computing engine Spark, it calls the spark-submit (task submission instruction) command to start the job and establish a single Spark Driver unit; submits Yaml files of job descriptions to one or more K8s slave clusters to enable K8s slave clusters to establish slave cluster Spark executor units.
  • the spark-submit task submission instruction
  • Step S403 Receive the Spark driver unit registration information of the K8s main cluster to complete the registration of the Spark driver unit in the collaboration center, and enable K8s to send the registration information to the registered Spark driver unit from the cluster to complete the registration of the Spark executor unit from the cluster in the Spark driver.
  • the registration of the unit enables the registered slave cluster Spark executor unit to execute the jobs sent by the K8s master cluster under the configuration of the Spark driver unit;
  • the collaboration center While receiving the registration information of the Spark driver unit sent by the K8s main cluster, the collaboration center also sends a reverse proxy startup request to the K8s main cluster, and establishes the K8s slave cluster Spark executor unit through Feedback information to enable the Spark driver unit to determine the establishment information of the Spark executor unit of the K8s slave cluster, and activate the reverse proxy unit to enable the K8s slave cluster to send the address of the Spark executor unit of the K8s slave cluster. and credential information to connect to the reverse proxy unit of the K8s main cluster. If the connection is successful, the K8s slave cluster's Spark executor unit sends registration information to the Spark driver unit to complete the K8s slave cluster Spark execution. The driver unit is registered in the Spark driver unit.
  • Step S404 After the execution of the job is completed, release the resources occupied by the execution of the job.
  • the present disclosure provides a Spark collaborative computing device for multiple K8s clusters, including a job interface module, a job assignment module, and a resource interface module.
  • the job assignment module includes a K8s cluster designation module, a job description Yaml file delivery module, and a registration information delivery module. and resource release module.
  • the job interface module is used to obtain jobs submitted by users.
  • the K8s cluster designation module is used to designate the K8s master cluster and K8s slave cluster based on the job status information of the computing engine Spark submitted by the user.
  • the job description Yaml file delivery module sends the job description Yaml to the K8s API interface of the K8s main cluster. file so that after the K8s master cluster receives the submission request from the computing engine Spark, it calls the spark-submit command to start the job and establish a single Spark drive unit; submit the Yaml file of the job description to multiple K8s slave clusters so that the K8s slave clusters can be established Spark executor unit from cluster.
  • the registration information delivery module receives the Spark driver unit registration information sent by the K8s main cluster to complete the registration of the Spark driver unit in the collaboration center, and enables K8s to send registration information from the cluster to the registered Spark driver unit to complete the Spark executor from the cluster.
  • the registration of the unit in the Spark driver unit enables the registered slave cluster Spark executor unit to execute the jobs sent by the K8s master cluster under the configuration of the Spark driver unit;
  • the collaboration center While receiving the registration information of the Spark driver unit sent by the K8s main cluster, the collaboration center also sends a reverse proxy startup request to the K8s main cluster, and establishes the K8s slave cluster Spark executor unit through the Feedback information to enable the Spark driver unit to determine the establishment information of the Spark executor unit of the K8s slave cluster, and take effect on the reverse proxy unit to enable the K8s slave cluster to send the address of the K8s Spark executor unit from the cluster and Credential information to connect to the reverse proxy unit of the K8s master cluster. If the connection is successful, registration information is sent to the Spark driver unit to complete the installation of the Spark executor unit of the K8s slave cluster in the Spark driver unit. registration.
  • the resource release module is used to release the resources occupied by the job after the job execution is completed.
  • the resource interface module is used to obtain the resource status of each cluster.
  • This disclosure provides Spark collaborative computing, operation methods and devices for multiple K8s clusters.
  • the computing resources of multiple different Kubernetes slave clusters are used to complete computing tasks.
  • this disclosure does not generate multiple Each drive unit realizes the resources of multiple clusters and makes Spark's computing process transparent; thus enabling the computing power of multiple clusters to effectively collaborate together to process complex Spark computing tasks.
  • FIG1a is an architecture diagram of a Spark collaborative operation method for multiple K8s clusters in an embodiment of the present disclosure.
  • Figure 1b is a flow chart of a Spark collaborative working method for multiple K8s clusters of the present disclosure.
  • Figure 2 is a schematic structural diagram of a Spark collaborative working device for multiple K8s clusters in an embodiment of the present disclosure.
  • Figure 3 is a flow chart of a multi-cluster collaborative work method of the present disclosure applied to a collaboration center.
  • FIG4 is a schematic diagram of the structure of a multi-cluster collaborative operation device disclosed in the present invention.
  • Figure 5 is a flow chart of the multi-cluster collaborative work method of the present disclosure applied to the main cluster.
  • Figure 6 is a flow chart of a multi-cluster collaborative work method of the present disclosure applied to slave clusters.
  • Figure 7 is a schematic structural diagram of a multi-cluster collaborative work device according to the present disclosure.
  • this disclosure provides a Spark collaborative working method for multiple K8s clusters. It is an Operator tool built based on Kubernetes API (Application Programming Interface) resources to manage and monitor deployment applications. Operator can be seen as a pattern to solve the problem of complex application containerization. Using Operator to customize resource management applications and their components, users can create, configure, and manage complex stateful applications. Operator follows the design concept of Kubernetes declarative API and Controller and is used to extend Kubernetes API. Operators are built on Kubernetes' resource and controller concepts, but also include Spark-specific domain knowledge.
  • Kubernetes API Application Programming Interface
  • the Spark collaborative work method for multiple K8s clusters includes the establishment of cross-cluster communication tunnels, the creation of the main cluster Driver Pod, the creation of the slave cluster Executor Pod, the registration of the slave cluster Executor Pod, and the master cluster Driver Pod task.
  • Distribution and other processes among which, in Figure 1a, C represents the collaboration center, which can be implemented through one or more computers, M represents the K8S main cluster or K8S single cluster, S represents the K8S slave cluster, and the K8S main cluster can be implemented through one or more computers.
  • K8S single cluster can be realized by one or more computers, and K8S slave cluster can be realized by one or more computers.
  • the Spark collaborative working method for multiple K8s clusters provided by the embodiment of the present disclosure may include the following steps.
  • Step S1 Submit the user's Spark application to the collaboration center.
  • Step S2 The collaboration center determines whether it is completed by a K8S single cluster based on the user-selected strategy and status information such as computing resources and data storage. If so, the collaboration center selects an appropriate K8S single cluster to deploy the current Spark application; if multiple clusters are required When the collaboration is completed, jump to step S3.
  • Step S3 The collaboration center designates a K8S master cluster and one or more K8S slave clusters based on the user-selected policies, computing resources, and data storage status.
  • step S3 may include establishing a cross-cluster communication tunnel.
  • a routing table mechanism and a VxLan can be used to build a cross-cluster communication tunnel.
  • VxLan Virtual eXtensible LAN, scalable virtual local area network
  • VTEP VXLAN Tunnel End Point
  • VNI Virtual Network Infrastructure, virtual network infrastructure
  • Step S4 The collaboration center submits the Yaml (YAML Ain't Markup Language) file of the job description to the main cluster API (Application Programming Interface) Server.
  • the main cluster receives Spark's submission request, calls spark-submit to start the job, and establishes the Spark Driver class.
  • the main cluster sends the address information and certificate information of the two types of Pods to the collaboration center; in Figure 1a, two types of Spark Pods are established, namely, Spark Driver class Pods and Spark Executor Pods, that is, the main cluster establishes the driver unit and Main cluster executor unit; Pod is the smallest unit of k8s, and containers are included in Pods, that is, Pod is a collection of containers.
  • the creation of the main cluster Driver Pod is based on the k8s client (client) mechanism.
  • the client communicates with the main cluster API Server and inputs parameters to complete the creation process of the Driver Pod.
  • the client is actually the client of HTTPS (Hypertext Transfer Protocol Secure).
  • HTTPS Hypertext Transfer Protocol Secure
  • the API Server is actually responsible for Pod creation, deletion, container creation, etc. If the controller wants to complete these actions, it needs to create HTTPS , and sends a request to the API Server; the process of creating a k8s-based client client mechanism is as follows:
  • Step S5 Receive job description information (described in a Yaml file) from the cluster, establish Spark Executor Pods; send an approval (or rejection) response to the collaboration center.
  • the creation process of the slave cluster Executor Pod is similar to the creation process of the main cluster Driver Pod. Obtain the configuration information and container of the Executor Pod from the collaboration center and create the corresponding Executor Pod.
  • Step S6 The collaboration center receives the Spark Driver Pod registration information sent by the main cluster and sends a request to start the Reverse Proxy.
  • the collaboration center sends the registration information of the Spark Driver Pod to the main cluster to complete the registration of the Spark Driver Pod in the main cluster and/or the collaboration center.
  • the Driver Pod determines the number of Executors for this job.
  • Reverse Proxy takes effect through the deployed Reverse Proxy.
  • Step S7 Send the address information and certificate information from the cluster to the main cluster to connect with the Reverse Proxy of the main cluster. If the connection is successful, the slave cluster sends the registration information of the Executor Pods to the main cluster and sends a heartbeat message to the main cluster.
  • Step S8 The main cluster determines whether it has received the registration information and heartbeat messages of all Executor Pods. If it is received, the connection is successful and jumps to step S9; if it is not received, it continues to wait.
  • registration from the cluster Executor container includes:
  • the container sends a heartbeat message to confirm the establishment of a communication link with the Driver Pod.
  • the access credentials are provided to the Driver Pod to complete the registration of Executor Pods in the Driver Pod.
  • Step S9 The master cluster sends Tasks to all Executor Pods in the slave cluster.
  • main cluster Driver container task assignment and scheduling include:
  • the driver of the master cluster After the slave cluster executor unit successfully registers, the driver of the master cluster enters the task assignment process. Since the data stored in different clusters are different, tasks are assigned and scheduled based on the storage location of the data as much as possible.
  • Step S10 The master cluster and the slave cluster start to execute the job under the driver configuration. If the job execution is completed, the collaboration center notifies the resource to be released.
  • a Spark collaborative computing device for multiple K8s clusters includes a job interface module, a job assignment module and a resource interface module.
  • the job assignment module includes a K8s cluster designation module, a job description YAML file delivery module, a registration information delivery module and a resource release module.
  • Job interface module used to obtain jobs submitted by users.
  • the K8s cluster designation module is used to specify K8s based on the job status information of the computing engine Spark submitted by the user. Master cluster and K8s slave cluster.
  • the job description Yaml file delivery module is used to send the Yaml file of the job description to the K8s API interface of the K8s main cluster, so that after the K8s main cluster receives the submission request from the computing engine Spark, it calls the spark-submit command to start the job and create a single
  • the main cluster Spark driver unit and a set of main cluster Spark executor units feedback the Spark driver unit and Spark executor unit addresses; submit Yaml files of job descriptions to multiple K8s slave clusters to enable K8s slave clusters to establish slave cluster Spark executors. unit and feedback acceptance or rejection.
  • a single Spark Driver is implemented to use the computing resources of different Kubernetes clusters to complete computing tasks. Different from the implementation method based on Kubernetes Fed, this method does not generate multiple Spark Drivers, making the resources of multiple clusters transparent to the Spark computing process;
  • Figure 2 Kube Proxy is the network proxy of the Kubernetes container scheduling and orchestration platform.
  • the registration information delivery module receives the registration information of the Spark driver unit of the K8s main cluster to complete the registration of the Spark driver unit in the collaboration center, and enables K8s to send registration information from the cluster to the registered Spark driver unit to complete the Spark executor from the cluster.
  • the registration of the unit in the Spark driver unit enables the registered slave cluster Spark executor unit to execute the jobs sent by the K8s master cluster under the configuration of the Spark driver unit.
  • the resource release module releases the resources occupied by the job after the job execution is completed.
  • the resource interface module is used to obtain the resource status of each cluster.
  • a multi-cluster collaborative operation method is applied to the collaboration center and includes the following steps.
  • Step S101 Specify the primary cluster and the secondary cluster based on the job status information submitted by the user.
  • a job refers to a collection of work that the user requires the computing system to do during a solution or transaction processing process. It includes user programs, required data set control commands, etc.
  • a job is composed of a series of ordered steps. Executing a job will involve computing, storage, etc. on multiple network nodes; job status information includes user-selected policies and status information such as computing resources and data storage.
  • the collaboration center determines whether it is a single-cluster job based on the job status information submitted by the user. If so, it selects a single cluster to execute the service and releases the occupied resources after the execution is completed. Otherwise, the master cluster and the slave cluster are designated and step S102 is performed.
  • Step S102 Send the job description file to the master cluster so that the master cluster can create a driver unit; submit the job description file to the slave cluster so that the slave cluster can create a group of slave cluster executor units.
  • a group of slave cluster executor units includes one or more slave cluster executor units.
  • a group of slave cluster executor units includes multiple slave cluster executor units, and the multiple slave cluster executor units may be established by one or more slave clusters.
  • the master cluster executor unit After submitting the job description file to the master cluster so that the master cluster starts the job, the master cluster executor unit is also established.
  • Step S103 Receive the registration information of the driver unit of the main cluster to complete the registration of the driver unit in the collaboration center, and enable the slave cluster executor unit to send the registration information to the registered driver unit to complete the slave cluster executor unit.
  • the registration of the element in the driver unit enables the registered slave cluster executor unit to execute the job sent by the master cluster.
  • the main cluster executor unit After receiving the registration information of the driver unit of the main cluster to complete the registration of the driver unit in the collaboration center, the main cluster executor unit also sends the registration information to the registered driver unit to complete the registration of the main cluster executor unit in the driver unit, Let the registered main cluster executor unit execute the job sent by the main cluster.
  • the collaboration center While the collaboration center sends the registration information of the driver unit to the main cluster, it also sends a reverse proxy (Reverse Proxy in Figure 1a) startup request to the main cluster, and uses the feedback information established by the slave cluster executor unit to enable the driver unit to determine the slave cluster.
  • the establishment information of the cluster executor unit activates the reverse proxy unit, so that the slave cluster sends the address and credential information of the slave cluster executor unit to connect with the reverse proxy unit of the main cluster. If the connection is successful, the slave cluster executor The unit sends registration information to the driver unit to complete the registration of the slave cluster executor unit in the driver unit.
  • the slave cluster to send registration information to the registered drive unit while also sending heartbeat messages, so that the master cluster determines whether the connection is successful based on the registration information and heartbeat messages of the slave cluster. If both are received, the connection is successful and the job is sent. (Task in Figure 1a); otherwise, the connection fails and continues to wait.
  • Step S104 After the job is executed, the resources occupied by executing the job are released.
  • a collaborative operation device for multiple clusters includes a job interface module, a job assignment module and a resource interface module.
  • the job assignment module includes a cluster designation module, a job description file delivery module, a registration information delivery module and Resource release module.
  • Job interface module used to obtain jobs submitted by users.
  • the cluster designation module is used to designate the master cluster and slave cluster based on the job status information submitted by the user.
  • the job description file delivery module is used to send the job description file to the master cluster so that the master cluster can establish the drive unit; submit the job description file to the slave cluster so that the slave cluster can establish the slave cluster executor unit.
  • the registration information delivery module is used to receive the registration information of the driver unit of the main cluster to complete the registration of the driver unit in the collaboration center, and enable the slave cluster to send registration information to the registered driver unit to complete the registration of the slave cluster executor unit in the driver unit. Registration enables the registered slave cluster executor unit to execute jobs sent by the master cluster.
  • the resource release module is used to release the occupied resources after the job execution is completed.
  • the resource interface module is used to obtain the resource status of each cluster.
  • a collaborative work method for multiple clusters is applied to the main cluster and includes the following steps:
  • Step S201 Obtain the job description file sent by the collaboration center and create a driver unit.
  • Step S202 Send the registration information of the driver unit, complete the registration of the driver unit in the collaboration center, obtain the registration information sent from the cluster executor unit through the registered driver unit, and complete the registration of the slave cluster executor unit in the driver unit.
  • the slave cluster executor is created from the slave cluster based on the job description file of the collaboration center.
  • Step S203 Send the job to the registered slave cluster, so that the registered slave cluster executor unit executes the job.
  • a collaborative work method for multiple clusters, applied to slave clusters includes the following steps:
  • Step S301 Obtain the job description file sent by the collaboration center and establish the slave cluster executor unit.
  • Step S302 Send registration information to the registered driver unit to complete the registration of the slave cluster executor unit in the driver unit; the driver unit is established by the master cluster according to the job description file of the collaboration center; the registration of the driver unit is Registration completed at the collaboration center based on the registration information of the drive unit received by the collaboration center.
  • Step S303 Obtain the job sent by the main cluster and execute the job through the slave cluster executor unit.
  • the present disclosure also provides an embodiment of a multi-cluster-oriented collaborative working device.
  • an embodiment of the present disclosure provides a collaborative working device for multiple clusters, including a memory and one or more processors.
  • the memory stores executable code
  • the one or more processors execute the executable code.
  • When executing the code it is used to implement a multi-cluster-oriented collaborative work method in the above embodiment.
  • the embodiment of the present disclosure of a collaborative working device for multiple clusters can be applied to any device with data processing capabilities, and any device with data processing capabilities can be a device or device such as a computer.
  • the device embodiments may be implemented by software, or may be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running them through the processor of any device with data processing capabilities. From the hardware level, as shown in Figure 7, it is a hardware structure diagram of any device with data processing capabilities where the multi-cluster collaborative operation device of the present disclosure is located.
  • any device with data processing capabilities where the device in the embodiment is located may also include other hardware based on the actual functions of any device with data processing capabilities, which will not be described again. .
  • the device embodiment since it basically corresponds to the method embodiment, please refer to the partial description of the method embodiment for relevant details.
  • the device embodiments described above are only illustrative.
  • the units described as separate components may or may not be physically separated.
  • the components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. Persons of ordinary skill in the art can understand and implement the method without any creative effort.
  • Embodiments of the present disclosure also provide a computer-readable storage medium on which a program is stored.
  • a program is stored on which a program is stored.
  • the program is executed by a processor, a multi-cluster-oriented collaborative work method in the above embodiments is implemented.
  • the computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory.
  • the computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device.
  • the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability.
  • the computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Hardware Redundancy (AREA)

Abstract

本公开公开了面向多K8s集群的Spark协同计算、作业方法及装置，针对目前多K8s集群大多采用联邦集群的模式，而Spark自身的调度和优化的方法不能跨域实现的问题，通过实现跨域的作业方法，将多K8s集群设置为主集群和从集群，主集群负责创建Spark的Driver容器和Pod，从集群负责创建Spark的Executor容器和Pod。在创建容器之后，通过协同中心汇聚地址信息和访问凭证，在主集群和从集群之间建立直联的隧道，并通过隧道实现从集群中的容器向Driver注册并持续发送心跳报文。Driver在注册完成之后，可以通过隧道下发执行任务，并通过窄依赖/宽依赖等不同的数据依赖关系，实现算力在不同集群的最优调度。

Description

面向多K8s集群的Spark协同作业方法及装置 技术领域
本公开涉及云计算和大数据技术领域，尤其是涉及面向多K8s集群的Spark协同计算、作业方法及装置。
背景技术
Apache Spark是专为大规模数据处理而设计的快速通用的计算引擎，2009年，Spark诞生于加州大学伯克利分校的AMP实验室(the Algorithms,Machines and People lab)，并于2010年开源。2013年，Spark捐献给阿帕奇软件基金会(Apache Software Foundation)，并于2014年成为Apache顶级项目。如今，Spark成为了大大小小企业与研究机构的常用工具之一，Spark平台所累积的成熟应用已经成为数据行业重要的资产。
K8s(Kubernetes)是开源的容器调度编排平台，由于Kubernetes适用服务抽象支持命名和负载均衡，并且通过标签组织多容器调度单元Pod(Plain Old Documentation)，实现了较好的灵活性、可用性和负载均衡性。因此，在云原生(Cloud Native)技术发展的浪潮之中，Kubernetes伴随着容器技术的发展，成为了目前云时代的操作系统。Kubernetes作为容器编排领域的事实标准和云原生领域的关键项目，已经是云原生时代工程师最需要理解与实践的核心技术。
大数据、大模型的智能计算为代表的复杂应用的出现，使得单集群资源很难胜任超大型、复杂的计算任务，尤其是大数据相关的复杂计算任务，需要规模较大的数据中心。然而数据中心缺少弹性能力，为保证业务应对突发请求等情况，普遍存在资源浪费情况。目前的多Kubernetes集群多采用联邦集群的模式，Spark自身的调度和优化的方法不能跨域实现。
发明内容
为解决现有技术的不足，实现多集群的算力有效的协同在一起处理复杂Spark计算任务的目的，本公开采用如下的技术方案。
本公开提供一种面向多集群的协同作业方法，应用于协同中心，包括如下步骤。
步骤S101：根据用户提交的作业状态信息，指定主集群和从集群。
作业是指用户在一次解决或是一个事务处理过程中要求计算系统所做的工作的集合，它包括用户程序、所需要的数据集控制命令等。作业是由一系列有序的步骤组成的，在执行一个作业会涉及多个网络节点上的计算、存储等；作业状态信息包括用户选择的策 略和计算资源、数据存储等状态信息。
步骤S102：向主集群发送作业描述文件，以使主集群建立驱动器单元；向从集群提交作业描述文件，以使从集群建立从集群执行器单元。
步骤S103：接收主集群发送的驱动器单元的注册信息，以完成驱动器单元在协同中心的注册，使从集群向注册的驱动器单元发送注册信息，以完成从集群执行器单元在驱动器单元的注册，使注册的从集群执行器单元执行主集群发送的作业；
协同中心接收主集群发送的驱动器单元的注册信息的同时，还向主集群发送反向代理启动请求，并通过从集群执行器单元建立的反馈信息，使驱动器单元确定从集群执行器单元的建立信息，生效反向代理单元，以使从集群发送从集群执行器单元的地址和凭证信息，以与主集群的反向代理单元连接，若连接成功，则向驱动器单元发送注册信息，以完成从集群执行器单元在驱动器单元的注册。
步骤S104：作业执行完成后，释放执行作业所占用的资源。
进一步地，所述步骤S101中，协同中心根据用户提交的作业状态信息判断是否单集群作业，若是，则选择单集群执行业务，执行完成后释放占用的资源，否则指定主集群和从集群，并执行步骤S102。
进一步地，所述步骤S102中，向主集群发送作业描述文件，以使主集群启动作业后，还建立了主集群执行器单元；步骤S103中，接收主集群发送的驱动器单元的注册信息，以完成驱动器单元在协同中心的注册后，还使主集群执行器单元向注册的驱动器单元发送主集群执行器单元的注册信息，以完成主集群执行器单元在驱动器单元的注册，使注册的主集群执行器单元执行主集群发送的作业。
基于反向代理实现自定义协议，实现单一计算任务下，跨域集群的直接通信，该方法避免了通过第三方的转发，实现高效通信。
进一步地，所述步骤S103中，以使从集群向注册的驱动器单元发送注册信息的同时，还向主集群发送心跳报文，使得主集群根据从集群的注册信息和心跳报文判断连接是否成功，若主集群收到该组从集群执行器单元的注册信息和所述心跳报文，则连接成功，发送作业；否则，连接失败，继续等待。
本公开提供一种面向多集群的协同作业装置，包括作业接口模块、作业指派模块和资源接口模块，所述作业指派模块包括集群指定模块、作业描述文件下发模块、注册信息下发模块和资源释放模块。
作业接口模块用于获取用户提交的作业。
集群指定模块用于根据用户提交的作业状态信息，指定主集群和从集群。
作业描述文件下发模块用于向主集群发送作业描述文件，以使主集群建立驱动器单元；向从集群提交作业描述文件，以使从集群建立从集群执行器单元。
注册信息下发模块，用于向主集群发送驱动器单元注册信息，以完成驱动器单元在协同中心的注册，使从集群向注册的驱动器单元发送注册信息，以完成从集群执行器单元在驱动器单元的注册，使注册的从集群执行器单元执行主集群发送的作业。
协同中心向接收主集群发送的驱动器单元的注册信息的同时，还向主集群发送反向代理启动请求，并通过该组从集群执行器单元建立的反馈信息，使驱动器单元确定从集群执行器单元的建立信息，生效反向代理单元，以使从集群发送该组从集群执行器单元的地址和凭证信息至主集群，以与主集群的反向代理单元连接，若从集群与主集群的反向代理单元连接成功，则从集群向驱动器单元发送该组从集群执行器单元的注册信息，以完成该组从集群执行器单元在驱动器单元的注册。
资源释放模块用于作业执行完成后释放占用的资源。
资源接口模块用于获取各集群的资源状态。
本公开提供一种面向多集群的协同作业方法，应用于主集群，包括如下步骤。
步骤S201：获取协同中心发送的作业描述文件，建立驱动器单元。
步骤S202：向协同中心发送驱动器单元注册信息，完成驱动器单元在协同中心的注册，通过注册的驱动器单元，获取从集群执行器单元发送的注册信息，完成从集群执行器单元在驱动器单元的注册，从集群执行器，是从集群根据协同中心的作业描述文件建立的。
所述主集群向所述协同中心发送所述驱动器单元的注册信息的同时，还获取反向代理启动请求，通过该组从集群执行器单元建立的反馈信息，使所述驱动器单元确定该组从集群执行器单元的建立信息，生效反向代理单元，以使所述从集群发送该组从集群执行器单元的地址和凭证信息，以与主集群的反向代理单元连接，若连接成功，则所述从集群向所述驱动器单元发送该组从集群执行器单元的注册信息，以完成该组从集群执行器单元在所述驱动器单元的注册。
步骤S203：向注册的从集群发送作业，以使注册的从集群执行器单元执行作业。
本公开提供一种面向多集群的协同作业方法，应用于从集群，包括如下步骤。
步骤S301：获取协同中心发送的作业描述文件，建立从集群执行器单元；
步骤S302：向注册的驱动器单元发送注册信息，完成从集群执行器单元在驱动器单元的注册；驱动器单元，是主集群根据协同中心的作业描述文件建立的；驱动器单元的注册，是根据协同中心接收主集群发送的驱动器单元注册信息，在协同中心完成的注册。
步骤S303：获取主集群发送的作业，并通过从集群执行器单元执行作业。
本公开提供一种面向多K8s集群的Spark协同计算方法，应用于协同中心，包括如下步骤。
步骤S401：根据用户提交的计算引擎Spark的作业状态信息，指定K8s主集群和K8s从集群。
步骤S402：向K8s主集群的K8s API接口发送作业描述的Yaml文件，以使K8s主集群收到计算引擎Spark的提交请求后，调用spark-submit(为任务提交指令)指令启动作业，建立单个Spark驱动器单元；向一个或者多个K8s从集群提交作业描述的Yaml文件，以使K8s从集群建立从集群Spark执行器单元。
实现单一Spark Driver利用不同Kubernetes集群的计算资源完成计算任务，不同于基于Kubernetes Fed的实现方法，本方法不会产生多个Spark Driver，能够实现多集群的资源对Spark的计算流程透明化。
步骤S403：接收K8s主集群的Spark驱动器单元注册信息，以完成Spark驱动器单元在协同中心的注册，使K8s从集群向注册的Spark驱动器单元发送注册信息，以完成从集群Spark执行器单元在Spark驱动器单元的注册，使注册的从集群Spark执行器单元，在Spark驱动器单元的配置下，执行K8s主集群发送的作业；
所述协同中心接收所述K8s主集群发送的所述Spark驱动器单元的注册信息的同时，还向所述K8s主集群发送反向代理启动请求，并通过所述K8s从集群Spark执行器单元建立的反馈信息，使所述Spark驱动器单元确定所述K8s从集群的Spark执行器单元的建立信息，生效反向代理单元，以使所述K8s从集群发送所述K8s从集群的Spark执行器单元的地址和凭证信息，以与K8s主集群的反向代理单元连接，若连接成功，则所述K8s从集群的Spark执行器单元向所述Spark驱动器单元发送注册信息，以完成所述K8s从集群Spark执行器单元在所述Spark驱动器单元的注册。
步骤S404：作业执行完成后，释放执行作业所占用的资源。
本公开提供一种面向多K8s集群的Spark协同计算装置，包括作业接口模块、作业指派模块和资源接口模块，作业指派模块包括K8s集群指定模块、作业描述Yaml文件下发模块、注册信息下发模块和资源释放模块。
作业接口模块用于获取用户提交的作业。
K8s集群指定模块用于根据用户提交的计算引擎Spark的作业状态信息，指定K8s主集群和K8s从集群。
作业描述Yaml文件下发模块，向K8s主集群的K8s API接口发送作业描述的Yaml 文件，以使K8s主集群收到计算引擎Spark的提交请求后，调用spark-submit指令启动作业，建立单个Spark驱动器单元；向多个K8s从集群提交作业描述的Yaml文件，以使K8s从集群建立从集群Spark执行器单元。
实现单一Spark Driver利用不同Kubernetes集群的计算资源完成计算任务，不同于基于Kubernetes Fed的实现方法，本方法不会产生多个Spark Driver，并能够实现多集群的资源对Spark的计算流程透明化。
注册信息下发模块，接收K8s主集群发送的Spark驱动器单元注册信息，以完成Spark驱动器单元在协同中心的注册，使K8s从集群向注册的Spark驱动器单元发送注册信息，以完成从集群Spark执行器单元在Spark驱动器单元的注册，使注册的从集群Spark执行器单元，在Spark驱动器单元的配置下，执行K8s主集群发送的作业；
所述协同中心接收所述K8s主集群发送的所述Spark驱动器单元的注册信息的同时，还向所述K8s主集群发送反向代理启动请求，并通过所述K8s从集群Spark执行器单元建立的反馈信息，使所述Spark驱动器单元确定所述K8s从集群的Spark执行器单元的建立信息，生效反向代理单元，以使所述K8s从集群发送所述K8s从集群Spark执行器单元的地址和凭证信息，以与所述K8s主集群的反向代理单元连接，若连接成功，则向所述Spark驱动器单元发送注册信息，以完成所述K8s从集群的Spark执行器单元在所述Spark驱动器单元的注册。
资源释放模块用于在作业执行完成后，释放执行作业所占用的资源。
资源接口模块用于获取各集群的资源状态。
本公开的优势和有益效果在于：
本公开提供面向多K8s集群的Spark协同计算、作业方法及装置，通过单一Spark驱动器单元，利用多个不同Kubernetes从集群的计算资源，完成计算任务，不同于现有技术，本公开不会产生多个驱动器单元，实现多集群的资源，对Spark的计算流程透明化；从而实现了多集群的算力有效的协同在一起处理复杂Spark计算任务。
附图说明
图1a是本公开实施例中一种面向多K8s集群的Spark协同作业方法的架构图。
图1b是本公开的一种面向多K8s集群的Spark协同作业方法的流程图。
图2是本公开实施例中一种面向多K8s集群的Spark协同作业装置的结构示意图。
图3是本公开的一种面向多集群的协同作业方法应用于协同中心的流程图。
图4是本公开的一种面向多集群的协同作业装置的结构示意图。
图5是本公开的一种面向多集群的协同作业方法应用于主集群的流程图。
图6是本公开的一种面向多集群的协同作业方法应用于从集群的流程图。
图7是本公开的一种面向多集群的协同作业设备的结构示意图。
具体实施方式
以下结合附图对本公开的具体实施方式进行详细说明。应当理解的是，此处所描述的具体实施方式仅用于说明和解释本公开，并不用于限制本公开。
如图1a、图1b所示，本公开提供的一种面向多K8s集群的Spark协同作业方法，是基于Kubernetes API(Application Programming Interface)资源构建的Operator工具，管理和监控部署应用程序。Operator可以看作是一种解决复杂应用容器化问题的模式。利用Operator定制资源管理应用及其组件，用户可以对复杂的有状态应用，进行创建、配置和管理。Operator遵循Kubernetes声明式API和Controller的设计理念，被用来扩展Kubernetes API。Operator基于Kubernetes的资源和控制器概念之上构建，但同时又包含了Spark特定的领域知识。本公开实施例提供的面向多K8s集群的Spark协同作业方法包括跨集群的通信隧道的建立、主集群Driver Pod的创立、从集群Executor Pod的创立、从集群Executor Pod的注册、主集群Driver Pod任务下发等过程，其中，在图1a中，C表示协同中心，可以通过一个或多个计算机实现，M表示K8S主集群或K8S单集群，S表示K8S从集群，K8S主集群可通过一个或多个计算机实现，K8S单集群可通过一个或多个计算机实现，K8S从集群可通过一个或多个计算机实现。本公开实施例提供的面向多K8s集群的Spark协同作业方法可以包括如下步骤。
步骤S1：将用户的Spark应用提交到协同中心。
步骤S2：协同中心根据用户选择的策略和计算资源、数据存储等状态信息，判断是否为K8S单集群完成，如果是，则协同中心选择合适的K8S单集群部署当前的Spark应用；如果需要多集群协同完成，则跳转到步骤S3。
步骤S3：协同中心根据用户选择的策略、计算资源、数据存储的状态，指定1个K8S主集群、1个或多个K8S从集群。
具体地，步骤S3可以包括跨集群通信隧道建立。
在一些实施例中，可采用路由表机制和VxLan(Virtual eXtensible LAN，可扩展虚拟局域网络)机制构建跨集群通信隧道。通过在Pod构建VTEP(VXLAN Tunnel End Point)也称为VxLan隧道的起点和终点，通过VNI(Virtual Network Infrastructure，虚拟网络基础设施)构建了不同用户的封装机制，实现了安全通信。基于VxLan发送报文的过程 如下所示：
开始
输入：需要跨域发送的消息Msg
输出：创建成功或失败状态
SetIPTableRule(TargetIP)#设置路由表规则，报文到达VTEP起点
Packet＝SetupIpOverIP(Msg)#构建IP over IP报文
SendPacket(Package)#发送到目的集群VTEP终点
Msg＝GetOriginalIP(Package)#恢复IP报文，发送给容器
Status＝GetStatus()#获取通信状态
结束
步骤S4：协同中心向主集群API(Application Programming Interface)Server提交作业描述的Yaml(YAML Ain't Markup Language)文件，主集群收到Spark的提交请求，调用spark-submit启动作业，建立Spark Driver类Pod和Spark Executor的Pods；主集群向协同中心发送两类Pods的地址信息、证书信息；图1a中建立两类Spark Pods即建立Spark Driver类Pod和Spark Executor的Pods，即主集群建立驱动器单元和主集群执行器单元；Pod是k8s的最小单元，容器包含在Pod中，即Pod是一组容器的集合。
针对主集群Driver Pod的创立，主集群Driver Pod的创建基于k8s的客户端(client)的机制，client与主集群API Server进行通信，输入参数完成Driver Pod的创建过程。client实际上是HTTPS(Hypertext Transfer Protocol Secure)的客户端，对于Pod创建、删除、容器创建等实际上都是由API Server来负责的，如果控制器(controller)想要完成这些动作，需要创建HTTPS，并向API Server发送请求；创建基于k8s的客户端client机制的过程如下所示：
步骤S5：从集群收到作业描述信息(以Yaml文件描述)，建立Spark Executor Pods；将同意(或拒绝)应答发送给协同中心。
从集群Executor Pod的创立过程类似于主集群Driver Pod创立过程。从集群从协同中心获取Executor Pod的配置信息和容器，创建相应的Executor Pod。
步骤S6：协同中心接收主集群发送的Spark Driver Pod的注册信息，并发送启动Reverse Proxy的请求。协同中心向主集群发送Spark Driver Pod的注册信息，以完成Spark Driver Pod在主集群和/或协同中心的注册。
此时Driver Pod确定了此次作业的Executors的数量。通过已经部署的Reverse Proxy，生效Reverse Proxy。
步骤S7：从集群向主集群发送地址信息和证书信息，以与主集群的Reverse Proxy连接，如连接成功，从集群向主集群发送Executor Pods的注册信息，并且向主集群发送心跳报文。
步骤S8：主集群判断是否收到所有Executor Pods的注册信息和心跳报文，如果收到，则连接成功，跳转步骤S9；如果未收到，则继续等待。
具体地，从集群Executor容器注册包括：
在从集群Executor Pods创建成功后，从集群Executor Pods中的容器通过向Driver
容器发送heartbeat报文，确认和Driver Pod建立通信链路。发送容器的资源信息和
访问凭证提供给Driver Pod，完成Executor Pods在Driver Pod的注册。

步骤S9：主集群向从集群的所有Executor Pods发送Task。
具体地，主集群Driver容器任务指派和调度包括：
在从集群执行器单元注册成功后，主集群的Driver进入指派任务task流程。由于不同集群所存储的数据存在差异，尽量将任务基于数据的存储位置进行指派和调度。
步骤S10：主集群和从集群开始在Driver的配置下执行作业，如果作业执行完成，协同中心通知释放资源。
如图2所示，一种面向多K8s集群的Spark协同计算装置，包括作业接口模块、作业指派模块和资源接口模块，作业指派模块包括K8s集群指定模块、作业描述Yaml文件下发模块、注册信息下发模块和资源释放模块。
作业接口模块，用于获取用户提交的作业。
K8s集群指定模块，用于根据用户提交的计算引擎Spark的作业状态信息，指定K8s 主集群和K8s从集群。
作业描述Yaml文件下发模块，用于向K8s主集群的K8s API接口发送作业描述的Yaml文件，以使K8s主集群收到计算引擎Spark的提交请求后，调用spark-submit指令启动作业，建立单个主集群Spark驱动器单元和一组主集群Spark执行器单元，反馈Spark驱动器单元和Spark执行器单元地址；向多个K8s从集群提交作业描述的Yaml文件，以使K8s从集群建立从集群Spark执行器单元，并反馈接收或拒绝。
实现单一Spark Driver利用不同Kubernetes集群的计算资源完成计算任务，不同于基于Kubernetes Fed的实现方法，本方法不会产生多个Spark Driver，实现多集群的资源对Spark的计算流程透明化；图2中的Kube Proxy即Kubernetes容器调度编排平台的网络代理。
注册信息下发模块，接收K8s主集群的Spark驱动器单元的注册信息，以完成Spark驱动器单元在协同中心的注册，使K8s从集群向注册的Spark驱动器单元发送注册信息，以完成从集群Spark执行器单元在Spark驱动器单元的注册，使注册的从集群Spark执行器单元，在Spark驱动器单元的配置下，执行K8s主集群发送的作业。
资源释放模块，作业执行完成后，释放执行作业所占用的资源。
资源接口模块，用于获取各集群的资源状态。
如图3所示，一种面向多集群的协同作业方法，应用于协同中心，包括如下步骤。
步骤S101：根据用户提交的作业状态信息，指定主集群和从集群。
作业是指用户在一次解决或是一个事务处理过程中要求计算系统所做的工作的集合，它包括用户程序、所需要的数据集控制命令等。作业是由一系列有序的步骤组成的，在执行一个作业会涉及多个网络节点上的计算、存储等；作业状态信息包括用户选择的策略和计算资源、数据存储等状态信息。
协同中心根据用户提交的作业状态信息判断是否单集群作业，若是，则选择单集群执行业务，执行完成后释放占用的资源，否则指定主集群和从集群，并执行步骤S102。
步骤S102：向主集群发送作业描述文件，以使主集群建立驱动器单元；向从集群提交作业描述文件，以使从集群建立一组从集群执行器单元。一组从集群执行器单元包括一个或者多个从集群执行器单元。可选地，一组从集群执行器单元包括多个从集群执行器单元，多个从集群执行器单元可以是一个或者多个从集群所建立的。
向主集群提交作业描述文件，以使主集群启动作业后，还建立了主集群执行器单元。
步骤S103：接收主集群的驱动器单元的注册信息，以完成驱动器单元在协同中心的注册，使从集群执行器单元向注册的驱动器单元发送注册信息，以完成从集群执行器单 元在驱动器单元的注册，使注册的从集群执行器单元执行主集群发送的作业。
接收主集群的驱动器单元的注册信息，以完成驱动器单元在协同中心的注册后，还使主集群执行器单元向注册的驱动器单元发送注册信息，以完成主集群执行器单元在驱动器单元的注册，使注册的主集群执行器单元执行主集群发送的作业。
协同中心向主集群发送驱动器单元的注册信息的同时，还向主集群发送反向代理(图1a中的Reverse Proxy)启动请求，并通过从集群执行器单元建立的反馈信息，使驱动器单元确定从集群执行器单元的建立信息，生效反向代理单元，以使从集群发送从集群执行器单元的地址和凭证信息，以与主集群的反向代理单元连接，若连接成功，则从集群执行器单元向驱动器单元发送注册信息，以完成从集群执行器单元在驱动器单元的注册。
基于反向代理实现自定义协议，实现单一计算任务下，跨域集群的直接通信，该方法通过避免第三方的转发，实现高效通信。
以使从集群向注册的驱动器单元发送注册信息的同时，还发送心跳报文，使得主集群根据从集群的注册信息和心跳报文判断连接是否成功，若均收到，则连接成功，发送作业(图1a中的Task)；否则，连接失败，继续等待。
步骤S104：作业执行完成后，释放执行作业所占用的资源。
这部分内容实施方式与上述方法实施例的实施方式类似，此处不再赘述。
如图4所示，一种面向多集群的协同作业装置，包括作业接口模块、作业指派模块和资源接口模块，作业指派模块包括集群指定模块、作业描述文件下发模块、注册信息下发模块和资源释放模块。
作业接口模块，用于获取用户提交的作业。
集群指定模块，用于根据用户提交的作业状态信息，指定主集群和从集群。
作业描述文件下发模块，用于向主集群发送作业描述文件，以使主集群建立驱动器单元；向从集群提交作业描述文件，以使从集群建立从集群执行器单元。
注册信息下发模块，用于接收主集群的驱动器单元的注册信息，以完成驱动器单元在协同中心的注册，使从集群向注册的驱动器单元发送注册信息，以完成从集群执行器单元在驱动器单元的注册，使注册的从集群执行器单元执行主集群发送的作业。
资源释放模块，用于作业执行完成后释放占用的资源。
资源接口模块，用于获取各集群的资源状态。
这部分内容实施方式与上述装置实施例的实施方式类似，此处不再赘述。
如图5所示，一种面向多集群的协同作业方法，应用于主集群，包括如下步骤：
步骤S201：获取协同中心发送的作业描述文件，建立驱动器单元。
步骤S202：发送驱动器单元的注册信息，完成驱动器单元在协同中心的注册，通过注册的驱动器单元，获取从集群执行器单元发送的注册信息，完成从集群执行器单元在驱动器单元的注册，所述从集群执行器，是从集群根据协同中心的作业描述文件建立的。
步骤S203：向注册的从集群发送作业，以使注册的从集群执行器单元执行作业。
这部分内容实施方式与上述方法实施例的实施方式类似，此处不再赘述。
如图6所示，一种面向多集群的协同作业方法，应用于从集群，包括如下步骤：
步骤S301：获取协同中心发送的作业描述文件，建立从集群执行器单元。
步骤S302：向注册的驱动器单元发送注册信息，完成从集群执行器单元在驱动器单元的注册；所述驱动器单元，是主集群根据协同中心的作业描述文件建立的；所述驱动器单元的注册，是根据协同中心接收的驱动器单元的注册信息，在协同中心完成的注册。
步骤S303：获取主集群发送的作业，并通过从集群执行器单元执行作业。
这部分内容实施方式与上述方法实施例的实施方式类似，此处不再赘述。
与前述一种面向多集群的协同作业方法的实施例相对应，本公开还提供了一种面向多集群的协同作业设备的实施例。
参见图7，本公开实施例提供的一种面向多集群的协同作业设备，包括存储器和一个或多个处理器，存储器中存储有可执行代码，所述一个或多个处理器执行所述可执行代码时，用于实现上述实施例中的一种面向多集群的协同作业方法。
本公开一种面向多集群的协同作业设备的实施例可以应用在任意具备数据处理能力的设备上，该任意具备数据处理能力的设备可以为诸如计算机等设备或装置。装置实施例可以通过软件实现，也可以通过硬件或者软硬件结合的方式实现。以软件实现为例，作为一个逻辑意义上的装置，是通过其所在任意具备数据处理能力的设备的处理器将非易失性存储器中对应的计算机程序指令读取到内存中运行形成的。从硬件层面而言，如图7所示，为本公开一种面向多集群的协同作业设备所在任意具备数据处理能力的设备的一种硬件结构图，除了图7所示的处理器、内存、网络接口、以及非易失性存储器之外，实施例中装置所在的任意具备数据处理能力的设备通常根据该任意具备数据处理能力的设备的实际功能，还可以包括其他硬件，对此不再赘述。
上述装置中各个单元的功能和作用的实现过程具体详见上述方法中对应步骤的实现过程，在此不再赘述。
对于装置实施例而言，由于其基本对应于方法实施例，所以相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本公开方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
本公开实施例还提供一种计算机可读存储介质，其上存储有程序，该程序被处理器执行时，实现上述实施例中的一种面向多集群的协同作业方法。
所述计算机可读存储介质可以是前述任一实施例所述的任意具备数据处理能力的设备的内部存储单元，例如硬盘或内存。所述计算机可读存储介质也可以是任意具备数据处理能力的设备的外部存储设备，例如所述设备上配备的插接式硬盘、智能存储卡(Smart Media Card，SMC)、SD卡、闪存卡(Flash Card)等。进一步的，所述计算机可读存储介质还可以既包括任意具备数据处理能力的设备的内部存储单元也包括外部存储设备。所述计算机可读存储介质用于存储所述计算机程序以及所述任意具备数据处理能力的设备所需的其他程序和数据，还可以用于暂时地存储已经输出或者将要输出的数据。
以上实施例仅用以说明本公开的技术方案，而非对其限制；尽管参照前述实施例对本公开进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本公开实施例技术方案的范围。

Claims (11)

  1. 一种面向多集群的协同作业方法，应用于协同中心，其特征在于，包括：
    步骤S101：根据用户提交的作业状态信息，指定主集群和从集群；
    步骤S102：向所述主集群发送作业描述文件，以使所述主集群建立单个驱动器单元；向所述从集群提交所述作业描述文件，以使所述从集群建立一组从集群执行器单元；
    步骤S103：向所述主集群发送所述驱动器单元的注册信息，以完成所述驱动器单元在所述协同中心的注册，使所述从集群向所述驱动器单元发送该组从集群执行器单元的注册信息，以完成该组从集群执行器单元在所述驱动器单元的注册，使该组从集群执行器单元执行所述主集群发送的作业；
    所述协同中心向所述主集群发送所述驱动器单元的注册信息的同时，还向所述主集群发送反向代理启动请求，并通过该组从集群执行器单元建立的反馈信息，使所述驱动器单元确定该组从集群执行器单元的建立信息，生效反向代理单元，以使所述从集群发送该组从集群执行器单元的地址和凭证信息，以与所述主集群的所述反向代理单元连接，若所述从集群与所述主集群的反向代理单元连接成功，则所述从集群向所述驱动器单元发送该组从集群执行器单元的注册信息，以完成该组从集群执行器单元在所述驱动器单元的注册；
    步骤S104：作业执行完成后，释放执行作业所占用的资源。
  2. 根据权利要求1所述的一种面向多集群的协同作业方法，其特征在于，所述步骤S101中，所述协同中心根据所述用户提交的作业状态信息判断是否单集群作业，若是，则选择单集群执行业务，执行完成后释放占用的资源，否则指定所述主集群和所述从集群，并执行步骤S102。
  3. 根据权利要求1所述的一种面向多集群的协同作业方法，其特征在于，所述步骤S102中，向所述主集群发送所述作业描述文件，以使所述主集群建立单个驱动器单元后，还建立了主集群执行器单元；步骤S103中，向所述主集群发送所述驱动器单元的注册信息，以完成所述驱动器单元在所述协同中心的注册后，还使所述主集群执行器单元向所述驱动器单元发送主集群执行器单元的注册信息，以完成所述主集群执行器单元在所述驱动器单元的注册，使所述主集群执行器单元执行所述主集群发送的所述作业。
  4. 根据权利要求1所述的一种面向多集群的协同作业方法，其特征在于，所述步骤S103中，以使从集群向所述驱动器单元发送该组从集群执行器单元的注册信息的同时，还向所述主集群发送心跳报文，使得主集群根据所述该组从集群执行器单元的注册信息和所述心跳报文判断连接是否成功，若所述主集群收到所述该组从集群执行器单元的注 册信息和所述心跳报文，则所述主集群和所述从集群的连接成功，所述主集群向所述从集群发送所述作业；否则，连接失败，继续等待。
  5. 一种面向多集群的协同作业装置，包括作业接口模块、作业指派模块和资源接口模块，其特征在于，所述作业指派模块包括集群指定模块、作业描述文件下发模块、注册信息下发模块和资源释放模块；
    所述作业接口模块，用于获取用户提交的作业；
    所述集群指定模块，用于根据用户提交的作业状态信息，指定主集群和从集群；
    所述作业描述文件下发模块，用于向所述主集群发送作业描述文件，以使所述主集群建立单个驱动器单元；向所述从集群提交所述作业描述文件，以使所述从集群建立一组从集群执行器单元；
    所述注册信息下发模块，用于向所述主集群发送所述驱动器单元的注册信息，以完成所述驱动器单元在所述协同中心的注册，使所述从集群向所述驱动器单元发送该组从执行器单元的注册信息，以完成所述从集群执行器单元在所述驱动器单元的注册，使所述从集群执行器单元执行所述主集群发送的作业；
    所述协同中心向所述主集群发送所述驱动器单元的注册信息的同时，还向所述主集群发送反向代理启动请求，并通过该组从集群执行器单元建立的反馈信息，使所述驱动器单元确定所述从集群执行器单元的建立信息，生效反向代理单元，以使所述从集群发送该组从集群执行器单元的地址和凭证信息至所述主集群，以与所述主集群的所述反向代理单元连接，若所述从集群与所述主集群的反向代理单元连接成功，则所述从集群向所述驱动器单元发送该组从集群执行器单元的注册信息，以完成该组从集群执行器单元在所述驱动器单元的注册；
    所述资源释放模块，用于作业执行完成后释放占用的资源；
    所述资源接口模块，用于获取各集群的资源状态。
  6. 一种面向多集群的协同作业方法，应用于主集群，其特征在于，包括：
    步骤S201：获取协同中心发送的作业描述文件，建立单个驱动器单元；
    步骤S202：获取所述驱动器单元的注册信息，完成所述驱动器单元在所述协同中心的注册，通过所述驱动器单元，获取一组从集群执行器单元发送的该组从集群执行器单元的注册信息，完成该组从集群执行器单元在所述驱动器单元的注册，该组从集群执行器单元，是所述从集群根据获取的所述协同中心的所述作业描述文件建立的；
    所述主集群获取所述驱动器单元的注册信息的同时，还获取反向代理启动请求，通过该组从集群执行器单元建立的反馈信息，使所述驱动器单元确定该组从集群执行器单 元的建立信息，生效反向代理单元，以使所述从集群发送该组从集群执行器单元的地址和凭证信息，以与主集群的反向代理单元连接，若连接成功，则所述从集群向所述驱动器单元发送该组从集群执行器单元的注册信息，以完成该组从集群执行器单元在所述驱动器单元的注册；
    步骤S203：向所述从集群发送作业，以使该组从集群执行器单元执行作业。
  7. 一种面向多集群的协同作业方法，应用于从集群，其特征在于，包括：
    步骤S301：获取协同中心发送的作业描述文件，建立一组从集群执行器单元；
    步骤S302：向驱动器单元发送注册信息，完成该组从集群执行器单元在驱动器单元的注册；所述驱动器单元，是主集群根据所述协同中心的所述作业描述文件建立的；所述驱动器单元的注册，是根据所述协同中心向所述主集群发送的驱动器单元注册信息，在协同中心完成的注册；
    该组从集群执行器单元基于所述协同中心向所述主集群发送反向代理启动请求，建立的反馈信息，使所述驱动器单元确定该组从集群执行器单元的建立信息，生效反向代理单元，从集群发送从集群执行器单元的地址和凭证信息，以与所述主集群的反向代理单元连接，若连接成功，则向所述驱动器单元发送注册信息，该组从集群执行器单元完成在所述驱动器单元的注册；
    步骤S303：获取所述主集群发送的作业，并通过该组从集群执行器单元执行作业。
  8. 一种面向多K8s集群的Spark协同计算方法，应用于协同中心，其特征在于，包括如下步骤：
    步骤S401：根据用户提交的计算引擎Spark的作业状态信息，指定K8s主集群和K8s从集群；
    步骤S402：向所述K8s主集群的K8s API接口发送作业描述的Yaml文件，以使所述K8s主集群收到计算引擎Spark的提交请求后，调用spark-submit指令启动作业，以使所述K8s主集群建立单个Spark驱动器单元；向多个K8s从集群提交作业描述的Yaml文件，以使所述K8s从集群建立从集群Spark执行器单元；
    步骤S403：向所述K8s主集群发送Spark驱动器单元的注册信息，以完成所述Spark驱动器单元在所述协同中心的注册，使所述从集群Spark执行器单元向所述Spark驱动器单元发送注册信息，以完成所述从集群Spark执行器单元在所述Spark驱动器单元的注册，使所述从集群Spark执行器单元，在所述Spark驱动器单元的配置下，执行所述K8s主集群发送的作业；
    所述协同中心向所述K8s主集群发送所述Spark驱动器单元的注册信息的同时，还 向所述K8s主集群发送反向代理启动请求，并通过所述K8s从集群Spark执行器单元建立的反馈信息，使所述Spark驱动器单元确定所述K8s从集群的Spark执行器单元的建立信息，生效反向代理单元，以使所述K8s从集群发送所述K8s从集群的Spark执行器单元的地址和凭证信息，以与K8s主集群的反向代理单元连接，若连接成功，则所述K8s从集群的Spark执行器单元向所述Spark驱动器单元发送注册信息，以完成所述K8s从集群Spark执行器单元在所述Spark驱动器单元的注册；以及
    步骤S404：所述作业执行完成后，释放执行所述作业所占用的资源。
  9. 一种面向多K8s集群的Spark协同计算装置，包括作业接口模块、作业指派模块和资源接口模块，其特征在于，所述作业指派模块包括K8s集群指定模块、作业描述Yaml文件下发模块、注册信息下发模块和资源释放模块；
    所述作业接口模块，用于获取用户提交的作业；
    所述K8s集群指定模块，用于根据用户提交的计算引擎Spark的作业状态信息，指定K8s主集群和K8s从集群；
    所述作业描述Yaml文件下发模块，用于向所述K8s主集群的K8s API接口发送作业描述的Yaml文件，以使K8s主集群收到计算引擎Spark的提交请求后，调用spark-submit指令启动作业，建立单个Spark驱动器单元；向多个K8s从集群提交作业描述的Yaml文件，以使所述多个K8s从集群建立多个从集群Spark执行器单元；
    所述注册信息下发模块，用于向所述K8s主集群发送Spark驱动器单元的注册信息，以完成所述Spark驱动器单元在协同中心的注册，使所述K8s从集群向所述Spark驱动器单元发送所述K8s从集群的Spark执行器单元的注册信息，以完成所述从集群Spark执行器单元在所述Spark驱动器单元的注册，使所述从集群Spark执行器单元，在Spark驱动器单元的配置下，执行所述K8s主集群发送的作业；
    所述协同中心向所述K8s主集群发送所述Spark驱动器单元的注册信息的同时，还向所述K8s主集群发送反向代理启动请求，并通过所述K8s从集群Spark执行器单元建立的反馈信息，使所述Spark驱动器单元确定所述K8s从集群的Spark执行器单元的建立信息，生效反向代理单元，以使所述K8s从集群发送所述K8s从集群Spark执行器单元的地址和凭证信息，以与所述K8s主集群的反向代理单元连接，若连接成功，则向所述Spark驱动器单元发送注册信息，以完成所述K8s从集群的Spark执行器单元在所述Spark驱动器单元的注册；
    所述资源释放模块，所述作业执行完成后，释放执行所述作业所占用的资源；
    所述资源接口模块，用于获取各集群的资源状态。
  10. 一种面向多集群的协同作业设备，其特征在于：包括存储器和一个或多个处理器，存储器中存储有可执行代码，所述一个或多个处理器执行所述可执行代码时，用于实现如权利要求1-4和6-8中任一项所述的方法。
  11. 一种计算机可读存储介质，其特征在于：其上存储有程序，该程序被处理器执行时，实现如权利要求1-4和6-8中任一项所述的方法。
PCT/CN2023/088148 2022-09-21 2023-04-13 面向多K8s集群的Spark协同作业方法及装置 Ceased WO2024060596A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/554,450 US11954525B1 (en) 2022-09-21 2023-04-13 Method and apparatus of executing collaborative job for spark faced to multiple K8s clusters

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211148298.1 2022-09-21
CN202211148298.1A CN115242877B (zh) 2022-09-21 2022-09-21 面向多K8s集群的Spark协同计算、作业方法及装置

Publications (1)

Publication Number Publication Date
WO2024060596A1 true WO2024060596A1 (zh) 2024-03-28

Family

ID=83681102

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/088148 Ceased WO2024060596A1 (zh) 2022-09-21 2023-04-13 面向多K8s集群的Spark协同作业方法及装置

Country Status (2)

Country Link
CN (1) CN115242877B (zh)
WO (1) WO2024060596A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119583553A (zh) * 2025-02-07 2025-03-07 浙江数新网络有限公司 Spark任务运行网关实现方法及系统

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115242877B (zh) * 2022-09-21 2023-01-24 之江实验室 面向多K8s集群的Spark协同计算、作业方法及装置
US11954525B1 (en) 2022-09-21 2024-04-09 Zhejiang Lab Method and apparatus of executing collaborative job for spark faced to multiple K8s clusters
CN115964130A (zh) * 2023-01-13 2023-04-14 中国联合网络通信集团有限公司 spark集群的部署方法和装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190250958A1 (en) * 2018-02-14 2019-08-15 Capital One Services, Llc Remotely managing execution of jobs in a cluster computing framework
CN113014625A (zh) * 2021-02-09 2021-06-22 华控清交信息科技(北京)有限公司 一种任务处理方法、装置和用于任务处理的装置
CN113835834A (zh) * 2021-09-10 2021-12-24 济南浪潮数据技术有限公司 一种基于k8s容器集群计算节点的扩容方法及系统
CN114942826A (zh) * 2022-05-20 2022-08-26 阿里巴巴（中国）有限公司 跨网络多集群系统及其访问方法及云计算设备
CN115242877A (zh) * 2022-09-21 2022-10-25 之江实验室 面向多K8s集群的Spark协同计算、作业方法及装置

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103713942B (zh) * 2012-09-28 2018-01-05 腾讯科技（深圳）有限公司 在集群中调度运行分布式计算框架的方法和系统
CN103744734B (zh) * 2013-12-24 2017-09-26 中国科学院深圳先进技术研究院 一种任务作业处理方法、装置及系统
US10552161B2 (en) * 2017-06-21 2020-02-04 International Business Machines Corporation Cluster graphical processing unit (GPU) resource sharing efficiency by directed acyclic graph (DAG) generation
US10671445B2 (en) * 2017-12-04 2020-06-02 Cisco Technology, Inc. Cost-optimal cluster configuration analytics package
US11329885B2 (en) * 2018-06-21 2022-05-10 International Business Machines Corporation Cluster creation using self-aware, self-joining cluster nodes
CN109033000A (zh) * 2018-08-14 2018-12-18 中国计量大学 一种弹性高效的光伏云计算集群控制系统及方法
CN110308984B (zh) * 2019-04-30 2022-01-07 北京航空航天大学 一种用于处理地理分布式数据的跨集群计算系统
CN110347489B (zh) * 2019-07-12 2021-08-03 之江实验室 一种基于Spark的多中心数据协同计算的流处理方法
CN113364727B (zh) * 2020-03-05 2023-04-18 北京金山云网络技术有限公司 容器集群系统、容器控制台和服务器
CN111767092B (zh) * 2020-06-30 2023-05-12 深圳前海微众银行股份有限公司 作业执行方法、装置、系统及计算机可读存储介质
CN115086312A (zh) * 2022-05-10 2022-09-20 兴业银行股份有限公司 实现kubernetes服务跨集群通信的方法及系统
CN115086330B (zh) * 2022-06-14 2024-03-01 亚信科技（中国）有限公司 跨集群负载均衡系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190250958A1 (en) * 2018-02-14 2019-08-15 Capital One Services, Llc Remotely managing execution of jobs in a cluster computing framework
CN113014625A (zh) * 2021-02-09 2021-06-22 华控清交信息科技(北京)有限公司 一种任务处理方法、装置和用于任务处理的装置
CN113835834A (zh) * 2021-09-10 2021-12-24 济南浪潮数据技术有限公司 一种基于k8s容器集群计算节点的扩容方法及系统
CN114942826A (zh) * 2022-05-20 2022-08-26 阿里巴巴（中国）有限公司 跨网络多集群系统及其访问方法及云计算设备
CN115242877A (zh) * 2022-09-21 2022-10-25 之江实验室 面向多K8s集群的Spark协同计算、作业方法及装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119583553A (zh) * 2025-02-07 2025-03-07 浙江数新网络有限公司 Spark任务运行网关实现方法及系统

Also Published As

Publication number Publication date
CN115242877A (zh) 2022-10-25
CN115242877B (zh) 2023-01-24

Similar Documents

Publication Publication Date Title
CN115242877B (zh) 面向多K8s集群的Spark协同计算、作业方法及装置
JP7724320B2 (ja) クロス・クラウド・オペレーションのためのクラウド・サービス
CN108737468B (zh) 云平台服务集群、构建方法及装置
JP6514687B2 (ja) ローカルまたは分散型コンピュータ・システムにおける柔軟なノード構成方法およびシステム
CN101188624B (zh) 基于虚拟机的网格中间件系统
US12301673B2 (en) Managed connectivity between cloud service edge locations used for latency-sensitive distributed applications
CN103634138B (zh) 分布式调度的远程管理与运维方法及其系统
WO2019218463A1 (zh) 基于Ansible工具自动搭建Kubernetes主节点的方法、装置、终端设备及可读存储介质
CN107087019A (zh) 一种端云协同计算架构及任务调度装置及方法
CN113821268A (zh) 一种与OpenStack Neutron融合的Kubernetes网络插件方法
WO2023186002A1 (zh) 一种资源调度方法、装置及设备
US11481228B2 (en) Self-service orchestration platform
CN107295096A (zh) 一种私有云计算服务平台、管理方法及系统
CN110806928A (zh) 一种作业提交方法及系统
CN114546648A (zh) 任务处理方法及任务处理平台
CN110377409A (zh) 一种任务处理方法、装置、设备和介质
CN110308987B (zh) 一种更新容器云上分布式训练任务连接参数的方法
JP6326062B2 (ja) 異なる環境どうし間でのジョブ実行依頼のトランスペアレントなルーティング
US11954525B1 (en) Method and apparatus of executing collaborative job for spark faced to multiple K8s clusters
CN117834704A (zh) 一云多芯应用的通信方法、装置、计算机设备及存储介质
CN105681087A (zh) 基于轻量级虚拟机的虚拟控制平面资源管理方法
CN118381822B (zh) 一种业务迁移方法、装置、系统、电子设备和存储介质
CN108073426A (zh) 一种基于云计算的软件管理方法、装置及系统
WO2024217338A1 (zh) 网关、网关热升级方法及系统
US12436806B2 (en) Global service catalog for provisioning services on a multi-cluster service mesh of a multi-cluster computing environment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23866893

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23866893

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 23866893

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 11/09/2025)

122 Ep: pct application non-entry in european phase

Ref document number: 23866893

Country of ref document: EP

Kind code of ref document: A1