CN117170812B - Numerical forecasting calculation cloud system based on research and development operation and maintenance integrated architecture - Google Patents

Abstract

The invention relates to a numerical forecasting calculation cloud system based on an integrated architecture of research and development operation and maintenance, which comprises the following components: a hybrid cluster comprising at least one hybrid node; the mirror image warehouse is used for storing a plurality of Docker basic mirrors and a plurality of numerical forecasting application mirrors; the conversion unit is used for synchronously converting the plurality of Docker application images into a plurality of Singularity application images; the node scheduling unit is used for receiving application research and development tasks, determining a first target node from the hybrid cluster, wherein the first target node is used for scheduling by the K8S scheduler, and the first target node is used for research and development of the numerical forecasting application; the node scheduling unit is also used for receiving the business job task, determining a second target node from the mixed cluster, wherein the second target node is used for Slurm for scheduling by a scheduler, and the second target node is used for pulling the numerical forecasting application and running the job, and has the advantages of integrating the research, development and operation environment of the numerical forecasting and improving the utilization rate of hardware resources.

Description

A numerical forecast computing cloud system based on R&D and operation and maintenance integrated architecture

Technical Field

The present invention relates to the field of data processing, and in particular to a numerical forecast computing cloud system based on a research and development and operation and maintenance integrated architecture.

Background technique

At present, the cluster scheduler of high-performance clusters used in the field of numerical forecasting is mainly slurm, the mainstream high-performance container is Singularity container, the cluster scheduler used for computing is mainly Kubernetes (abbreviated as K8S), and the mainstream container is Docker container. Container technology can realize the encapsulation and preservation of computing environment, rapid deployment, reuse and security isolation, and has achieved rapid development in recent years. Among them, Docker container with more mature technology and stronger isolation and its mainstream orchestration system K8S mainly focus on the production and management of containerized applications, which are suitable for numerical forecasting R&D system; while the lightweight and weakly isolated Singularity container and its mainstream job scheduling system Slurm mainly focus on the resource management and job scheduling of high-performance computing clusters, which are suitable for numerical forecasting business system. However, Slurm and K8S are not compatible at present, which makes it difficult for numerical forecasting business system to quickly deploy containerized applications created by R&D system, and the two cannot share underlying computing resources, which causes the separation of R&D environment and business environment and great waste of hardware computing resources.

Therefore, it is necessary to provide a numerical forecast computing cloud system based on an integrated R&D and operation and maintenance architecture to realize resource sharing and hybrid scheduling of Slrum nodes and K8S nodes, thereby integrating the R&D and operation and maintenance environment of numerical forecasting and improving the utilization of hardware resources.

Summary of the invention

One of the embodiments of the present specification provides a numerical forecast computing cloud system based on an integrated R&D and operation architecture, including: a hybrid cluster, including a Slurm cluster and a K8S cluster, the hybrid cluster including at least one Slurm node, at least one hybrid node and at least one K8S node, the hybrid node being scheduled by one of the Slurm cluster and the K8S cluster at the same time; an image warehouse, used to store multiple Docker basic images and multiple numerical forecast application images; a conversion unit, used to synchronously convert the multiple Docker application images into multiple Singularity application images; a shared storage unit, used to store the multiple Singularity application images converted by the conversion unit; a node scheduling unit, used to receive an application R&D task, and determine a first target node from the hybrid cluster, wherein the first target node is scheduled by the K8S scheduler, and the first target node is used for the R&D of numerical forecast applications; the node scheduling unit is also used to receive a business job task, and determine a second target node from the hybrid cluster, wherein the second target node is scheduled by the Slurm scheduler, and the second target node is used to pull the numerical forecast application and run the job.

In some embodiments, the first target node conducts research and development of numerical forecasting applications, including: the first target node schedules and allocates computing resources for performing the image production task; pulls the target Docker base image from the image warehouse; produces a numerical forecasting application image based on the target Docker base image and user instructions, and solidifies and uploads the produced numerical forecasting application image to the image warehouse.

In some embodiments, the second target node pulls the numerical forecast application and runs the job, including: the second target node schedules and allocates computing resources for performing the business job task; pulls the target Singularity application image from the shared storage unit; and runs the numerical forecast application based on the target Singularity application image and the numerical forecast task script.

In some embodiments, the multiple Docker base images include at least a MySQL application image, a programming language image, and an operating system image.

In some embodiments, the multiple numerical forecast application images include at least an HPL application image, an Fvcom application image, and a WRF application image.

In some embodiments, the node scheduling unit determines the first target node from the hybrid cluster, including: installing a node group priority plug-in on the Volcano scheduler; grouping the at least one hybrid node and the at least one K8S node according to resource type to generate multiple node groups, and configuring a priority for each of the node groups; the Volcano scheduler determines the first target node from the at least one hybrid node and the at least one K8S node based on the priority configuration of each of the node groups.

In some embodiments, the multiple node groups include at least a Slurm node group, a hybrid CPU node group, a hybrid GPU node group, a K8S CPU node group and a K8S GPU node group; the Volcano scheduler determines the first target node from the at least one hybrid node and the at least one K8S node based on the configuration priority of each of the node groups, including: determining whether the first target node exists in the K8S CPU node group; if the first target node does not exist in the K8S CPU node group, determining whether the first target node exists in the hybrid CPU node group; if the first target node does not exist in the hybrid CPU node group, determining whether the first target node exists in the K8S GPU node group; if the first target node does not exist in the K8S GPU node group, determining whether the first target node exists in the hybrid GPU node group.

In some embodiments, the node scheduling unit is further used to maintain a hybrid node list, wherein the hybrid node list is used to record the operation identification of each of the Slurm nodes, each of the hybrid nodes and each of the K8S nodes.

In some embodiments, the first target node is used at least for the development of numerical forecasting applications, management of development environments and resources, creation of numerical forecasting application images, and management of containers; the second target node is used at least for managing the operating environment, computing resources, operating results, and operating logs of the numerical forecasting application.

In some embodiments, the image creation task is initiated by a research and development user with root authority; and the business operation task is initiated by a business user without root authority.

BRIEF DESCRIPTION OF THE DRAWINGS

This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

FIG1 is a module diagram of a numerical forecast computing cloud system based on a research and development operation and maintenance integrated architecture according to some embodiments of this specification;

FIG2 is a schematic diagram of a structural diagram of a system architecture for integrating R&D and operation and maintenance environments according to some embodiments of this specification;

FIG3 is a schematic diagram of a process for developing and operating numerical forecasting applications according to some embodiments of the present specification;

FIG. 4 is a schematic diagram of a flow chart of determining a first target node from a hybrid cluster according to some embodiments of the present specification.

Detailed ways

In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, this specification can also be applied to other similar scenarios based on these drawings. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

It should be understood that the "system", "device", "unit" and/or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and/or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprises" and "includes" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may be added to these processes, or one or more operations may be removed from these processes.

First, the terms used in this specification are explained.

Numerical forecasting is an application of HPC. It uses large computers to perform numerical calculations to solve the basic equations of atmospheric motion, thereby predicting the state of atmospheric motion and weather phenomena at future times.

Kubernetes: is a portable, extensible, open source platform for managing containerized workloads and services that facilitates declarative configuration and automation.

Pod (container group): It is the smallest unit managed by Kubernetes. Multiple containers combined together are called a Pod.

Volcano: Volcano is the first and only Kubernetes-based container batch computing platform under CNCF, mainly used in high-performance computing scenarios. It provides a set of mechanisms that Kubernetes currently lacks, which are usually required for various high-performance workloads such as machine learning big data applications, scientific computing, special effects rendering, etc.

Volcano Job: VcJob for short, is a custom Job resource type of Volcano. Different from KubernetesJob, VcJob provides more advanced features, such as the ability to specify a scheduler, support a minimum number of running Pods, support Tasks, support lifecycle management, support for specified queues, support for priority scheduling, etc. Volcano Job is more suitable for high-performance computing scenarios such as machine learning, big data, and scientific computing.

CPU: The Central Processing Unit (CPU) is the computing and control core of the computer system and the final execution unit for information processing and program running.

GPU: Graphics processing unit (GPU), also known as display core, visual processor, display chip, is a microprocessor that specializes in performing image and graphics related calculations on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.).

Volcano Controller: Volcano controller, manages Volcano Job on the cluster.

Volcano Scheduler: Volcano Scheduler schedules VolcanoJob through a series of actions and plug-ins, and finds a most suitable node for it.

Slurm: The Slurm job scheduling tool is a free and open source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.

Singularity: Singularity is a container platform. It allows you to create and run containers that package software in a portable and repeatable way. You can build a container with Singularity on your laptop, and then run it on many of the world's largest HPC clusters, a local university or corporate cluster, a single server, in the cloud, or on a workstation down the hall.

Harbor is an enterprise-level image repository server for storing and distributing Docker images. It provides many practical functions such as permission management, log review, layered transmission, horizontal expansion, image replication, and graphical interface.

A container image is a read-only container packaged using Docker. Its content will not be changed after it is built. It can be considered as a standardized container template, and the container is a running instance of the image.

The numerical forecast computing cloud system based on the R&D and operation and maintenance integrated architecture can be used to realize the integrated system architecture of the R&D and operation and maintenance environment. FIG2 is a structural schematic diagram of the integrated system architecture of the R&D and operation and maintenance environment shown in some embodiments of this specification. As shown in FIG2, the integrated system architecture of the R&D and operation and maintenance environment includes two parts: the numerical forecast R&D system and the numerical forecast business system, which are respectively applied to the R&D scenario and the business scenario. In the numerical forecast R&D system, R&D users use K8S and Docker technologies to carry out the development of numerical forecast applications, the creation of application images, and the management of containers; in the numerical forecast business system, general business users use Slurm and Singularity technologies to pull the forecast application built by the R&D system and run the job. Containers are a virtualization technology that can provide isolated operating space for applications, and encapsulate, save and quickly transplant the system environment in which the program runs. Different containers share a system kernel, so compared with virtual machines, containers have the characteristics of less resource usage, shorter startup time, and more convenient migration and deployment.

The architecture simultaneously builds the K8S cluster in the numerical prediction R&D system and the Slurm cluster in the numerical prediction R&D system, and establishes a communication mechanism between the two to coordinate the scheduling of physical computing nodes and share the underlying hardware computing resources by dynamically acquiring and releasing node resources.

Docker in the numerical weather forecast R&D system in the architecture is currently the most widely used container technology, but it is not suitable for ordinary business users of numerical weather forecast who do not have root permissions.

Singularity in the numerical forecast business system in the architecture is simple, portable, easy to expand, easy to distribute, and user permissions are consistent inside and outside the container. It is more suitable for containerized deployment of numerical forecast applications than Docker, but lacks Docker's more mature community support and high-quality images, and cannot use efficient container orchestration systems such as K8S.

In some embodiments, in the numerical forecasting R&D system, the Volcano framework can be introduced on the basis of the K8S cluster to process numerical forecasting batch tasks. The Volcano framework enhances the job scheduling capability of K8S, can make up for the shortcomings of K8S scheduling capability, and supports a large number of mainstream computing frameworks in the fields of machine learning, deep learning, big data, etc.

Figure 1 is a module diagram of a numerical forecasting computing cloud system based on an integrated R&D and operation and maintenance architecture according to some embodiments of this specification. As shown in Figure 1, the numerical forecasting computing cloud system based on an integrated R&D and operation and maintenance architecture includes at least a hybrid cluster, a mirror warehouse, a conversion unit, a shared storage unit and a node scheduling unit.

A hybrid cluster may include a Slurm cluster and a K8S cluster. The hybrid cluster includes at least one Slurm node, at least one hybrid node and at least one K8S node. The hybrid node is scheduled by one of the Slurm cluster and the K8S cluster at the same time.

The image repository can be used to store multiple Docker base images and multiple numerical forecast application images. Among them, the multiple Docker base images at least include MySQL application images, programming language (e.g., Python, etc.) images, and operating system (e.g., Centos, etc.) images, and the multiple numerical forecast application images at least include HPL application images, Fvcom (Finite-Volume Coastal Ocean Model) application images, and WRF (Weather Research and Forecasting) application images.

The conversion unit can be used to synchronously convert multiple Docker application images into multiple Singularity application images.

The shared storage unit can be used to store multiple Singularity application images converted by the conversion unit. Multiple Singularity application images can be stored in the cloud platform shared storage system, and the shared storage system directory can be mounted to the instantiated container to achieve data persistence storage. The cloud platform shared storage system will be mounted to the physical cluster, and the Singularity application images (SIF files) on it can be instantiated by business users and executed through Slurm scheduling.

The node scheduling unit can be used to receive application development tasks and determine the first target node from the hybrid cluster, wherein the first target node is scheduled by the K8S scheduler. The first target node is at least used for the development of numerical forecasting applications, the management of development environments and resources, the creation of numerical forecasting application images, and the management of containers. The image production task can be initiated by a development user with root permissions.

The node scheduling unit is also used to receive business job tasks and determine a second target node from the hybrid cluster, wherein the second target node is scheduled by the Slurm scheduler, the second target node is used to pull the numerical forecast application and run the job, and the second target node is also used at least to manage the operating environment, computing resources, operating results and operating logs of the numerical forecast application. The business job task can be initiated by a business user who does not have root privileges.

FIG3 is a flow chart of developing and operating numerical forecast applications according to some embodiments of the present specification. As shown in FIG3 , in some embodiments, the first target node develops numerical forecast applications, including:

The first target node schedules and allocates computing resources for performing the image making task;

Pull the target Docker base image from the image repository;

A numerical forecast application image is created based on the target Docker base image and user instructions, and the created numerical forecast application image is solidified and uploaded to the image warehouse.

As shown in FIG3 , in some embodiments, the second target node pulls the numerical forecast application and runs the job, including:

The second target node schedules and allocates computing resources for performing business operation tasks;

Pull the target Singularity application image from the shared storage unit;

Run the numerical forecast application based on the target Singularity application image and numerical forecast task script.

In some embodiments, the node scheduling unit determines a first target node from the hybrid cluster, including:

Install the node group priority plugin on the Volcano scheduler;

Grouping at least one hybrid node and at least one K8S node according to resource types to generate multiple node groups, and configuring a priority for each node group, wherein the multiple node groups at least include a Slurm node group, a hybrid CPU node group, a hybrid GPU node group, a K8S CPU node group, and a K8S GPU node group;

The Volcano scheduler determines the first target node from at least one hybrid node and at least one K8S node based on the configuration priority of each node group.

FIG4 is a flow chart of determining a first target node from a hybrid cluster according to some embodiments of the present specification. As shown in FIG4 , in some embodiments, the Volcano scheduler determines the first target node from at least one hybrid node and at least one K8S node based on the configuration priority of each node group, including:

Determine whether the first target node exists in the K8S CPU node group;

If the first target node does not exist in the K8S CPU node group, determine whether the first target node exists in the hybrid CPU node group;

If the first target node does not exist in the hybrid CPU node group, determine whether the first target node exists in the K8S GPU node group;

If the first target node does not exist in the K8S GPU node group, determine whether the first target node exists in the hybrid GPU node group.

In some embodiments, the node scheduling unit is also used to maintain a hybrid node list, wherein the hybrid node list is used to record the operation identification of each Slurm node, each hybrid node and each K8S node. When the operation identification is idle, both types of cluster tasks can schedule the node to run. When a node runs a certain type of cluster task, the operation identification is set to only run this task. After the node's task is completed, the node's operation identification is reset to idle.

It can be understood that the numerical forecast computing cloud system based on the R&D and operation and maintenance integrated architecture can at least include the following beneficial effects:

1. It can realize the sharing and rapid construction of R&D environment, and reduce the time for multiple users to install and deploy numerical forecasting systems;

2. It can realize the packaging and preservation of the R&D environment, and realize the rapid recovery of the numerical forecasting R&D environment through the packaged files;

3. It can achieve efficient migration from the R&D environment to the operating environment, avoiding the problems of the traditional numerical forecasting system, such as complex environment, numerous dependencies, difficult deployment, and poor portability;

4. K8S and Slurm clusters communicate with each other, share underlying computing resources, and improve resource utilization efficiency;

5. Slurm directly schedules physical nodes instead of using multi-level scheduling strategies, greatly improving scheduling efficiency;

6. The container instance environment is directly scheduled and used by the Slurm physical cluster, using only one layer of containers, resulting in less container performance loss;

7. Use Singularity containers to package numerical forecast applications, further reducing container performance loss;

8. Implemented elastic scaling of the Slurm cluster, ensuring that cluster storage and computing resources can be dynamically scheduled according to user task requirements;

9. By running Slurm and K8S natively in a shared environment, the complexity and uncertainty of nested scheduling of Slurm and K8S are avoided, ensuring system stability.

The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and/or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims (8)

1. A numerical forecast computing cloud system based on an integrated R&D and operation and maintenance architecture, characterized by comprising: A hybrid cluster, comprising a Slurm cluster and a K8S cluster, wherein the hybrid cluster comprises at least one Slurm node, at least one hybrid node and at least one K8S node, and the hybrid node is scheduled by one of the Slurm cluster and the K8S cluster at the same time; Image repository, used to store multiple Docker base images and multiple numerical forecast application images; A conversion unit, configured to synchronously convert the multiple Docker application images into multiple Singularity application images; A shared storage unit, used to store the multiple Singularity application images converted by the conversion unit; A node scheduling unit, configured to receive an application development task and determine a first target node from the hybrid cluster, wherein the first target node is scheduled by the K8S scheduler and is used for the development of numerical forecasting applications; The node scheduling unit is further used to receive a business job task and determine a second target node from the hybrid cluster, wherein the second target node is scheduled by the Slurm scheduler, and the second target node is used to pull the numerical forecast application and run the job; The node scheduling unit determines a first target node from the hybrid cluster, including: Install the node group priority plugin on the Volcano scheduler; The at least one hybrid node and the at least one K8S node are grouped according to resource types to generate multiple node groups, and a priority is configured for each of the node groups; The Volcano scheduler determines the first target node from the at least one hybrid node and the at least one K8S node based on the configuration priority of each of the node groups; The multiple node groups include at least a Slurm node group, a hybrid CPU node group, a hybrid GPU node group, a K8S CPU node group and a K8S GPU node group; The Volcano scheduler determines the first target node from the at least one hybrid node and the at least one K8S node based on the configuration priority of each of the node groups, including: Determine whether the first target node exists in the K8S CPU node group; If the first target node does not exist in the K8S CPU node group, determine whether the first target node exists in the hybrid CPU node group; If the first target node does not exist in the hybrid CPU node group, determine whether the first target node exists in the K8S GPU node group; If the first target node does not exist in the K8S GPU node group, determine whether the first target node exists in the hybrid GPU node group. 2. According to claim 1, a numerical forecast computing cloud system based on an integrated R&D and operation architecture is characterized in that the first target node performs research and development of numerical forecast applications, including: The first target node schedules and allocates computing resources for performing the image making task; Pull the target Docker base image from the image repository; A numerical forecast application image is produced based on the target Docker base image and user instructions, and the produced numerical forecast application image is solidified and uploaded to the image warehouse. 3. A numerical forecast computing cloud system based on a research and development operation and maintenance integrated architecture according to claim 1, characterized in that the second target node pulls the numerical forecast application and runs the job, comprising: The second target node schedules and allocates computing resources for performing the business operation task; Pull the target Singularity application image from the shared storage unit; The numerical forecast application is run based on the target Singularity application image and the numerical forecast task script. 4. According to a numerical forecast computing cloud system based on a research and development and operation and maintenance integrated architecture as described in claim 1, it is characterized in that the multiple Docker basic images at least include a MySQL application image, a programming language image and an operating system image. 5. According to claim 4, a numerical forecast computing cloud system based on an integrated R&D and operation architecture is characterized in that the multiple numerical forecast application images at least include an HPL application image, an Fvcom application image and a WRF application image. 6. A numerical forecast computing cloud system based on a research and development and operation and maintenance integrated architecture according to any one of claims 1-5, characterized in that the node scheduling unit is also used to maintain a hybrid node list, wherein the hybrid node list is used to record the operation identification of each of the Slurm nodes, each of the hybrid nodes and each of the K8S nodes. 7. A numerical forecast computing cloud system based on a R&D and operation and maintenance integrated architecture according to any one of claims 1 to 5, characterized in that the first target node is at least used for the R&D of numerical forecast applications, the management of R&D environments and resources, the creation of numerical forecast application images, and the management of containers; The second target node is at least used to manage the operating environment, computing resources, operating results and operating log of the numerical forecast application. 8. A numerical forecast computing cloud system based on a research and development operation and maintenance integrated architecture according to any one of claims 1 to 5, characterized in that the image creation task is initiated by a research and development user with root authority; The business operation task is initiated by a business user who does not have root authority.