CN116775276A

CN116775276A - Distributed computing method, device, equipment and storage medium

Info

Publication number: CN116775276A
Application number: CN202310535030.1A
Authority: CN
Inventors: 魏曦明; 满远斌
Original assignee: Alibaba Damo Institute Hangzhou Technology Co Ltd
Current assignee: Alibaba Damo Institute Hangzhou Technology Co Ltd
Priority date: 2023-05-10
Filing date: 2023-05-10
Publication date: 2023-09-19

Abstract

The present disclosure relates to a distributed computing method, apparatus, device, and storage medium. The present disclosure provides for generating one or more logical execution plans from a grammar tree by retrieving code written by a user on a user interface and parsing the code into the grammar tree. And then, connecting each operator according to the dependency relationship among each operator in the logic execution plan aiming at each logic execution plan in one or more logic execution plans to form a calculation link corresponding to the logic execution plan, dividing the calculation link into a plurality of calculation tasks, distributing the plurality of calculation tasks to a plurality of distributed calculation nodes, and executing the calculation tasks in parallel to obtain a calculation result aiming at the target data. When the target data is a large amount of data, such as a large amount of remote sensing data, the processing and analysis processes of the large amount of remote sensing data are distributed to a plurality of distributed computing nodes to be executed in parallel by the method described in the embodiment, so that the efficiency of processing and analyzing the large amount of remote sensing data is greatly improved.

Description

Distributed computing method, device, equipment and storage medium

Technical Field

The disclosure relates to the field of information technology, and in particular, to a distributed computing method, a distributed computing device, and a storage medium.

Background

Currently, in some application scenarios, a large amount of data is generated. For example, in the remote sensing satellite industry, a large amount of remote sensing data is generated. Automated driving vehicles can generate a large amount of sensory data during drive tests.

However, the calculation process of the large amount of data is slow due to the large calculation amount of the large amount of data, and the efficiency of processing and analyzing the large amount of data is reduced.

Disclosure of Invention

In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a distributed computing method, apparatus, device and storage medium, so as to improve the efficiency of processing and analyzing the massive remote sensing data.

In a first aspect, an embodiment of the present disclosure provides a distributed computing method, including:

acquiring codes written by a user on a user interface, wherein the codes are used for describing calculation logic aiming at target data;

analyzing the codes into grammar trees, and generating one or more logic execution plans according to the grammar trees;

for each logic execution plan in the one or more logic execution plans, connecting each operator according to the dependency relationship among the operators in the logic execution plan to form a calculation link corresponding to the logic execution plan, and dividing the calculation link into a plurality of calculation tasks;

And distributing the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution to obtain a computing result aiming at the target data.

In a second aspect, embodiments of the present disclosure provide a distributed computing device comprising:

the acquisition module is used for acquiring codes written by a user on a user interface, wherein the codes are used for describing calculation logic aiming at target data;

the analysis module is used for analyzing the codes into grammar trees;

a generation module for generating one or more logic execution plans from the syntax tree;

the connection module is used for connecting each operator according to the dependency relationship among each operator in the logic execution plans aiming at each logic execution plan in the one or more logic execution plans to form a calculation link corresponding to the logic execution plan, and dividing the calculation link into a plurality of calculation tasks;

and the distribution module is used for distributing the plurality of calculation tasks to a plurality of distributed calculation nodes for parallel execution to obtain a calculation result aiming at the target data.

In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

a memory;

A processor; and

a computer program;

wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method according to the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having stored thereon a computer program for execution by a processor to implement the method of the first aspect.

The distributed computing method, the distributed computing device, the distributed computing equipment and the storage medium provided by the embodiment of the disclosure are characterized in that codes written on a user interface by a user are obtained, the codes are analyzed into grammar trees, and one or more logic execution plans are generated according to the grammar trees. And then, connecting each operator according to the dependency relationship among each operator in the logic execution plan aiming at each logic execution plan in one or more logic execution plans to form a calculation link corresponding to the logic execution plan, dividing the calculation link into a plurality of calculation tasks, distributing the plurality of calculation tasks to a plurality of distributed calculation nodes, and executing the calculation tasks in parallel to obtain a calculation result aiming at the target data. When the target data is a large amount of data, for example, a large amount of remote sensing data, the processing and analysis processes of the large amount of remote sensing data can be distributed to a plurality of distributed computing nodes to be executed in parallel by the method of the embodiment, so that the efficiency of processing and analyzing the large amount of remote sensing data is greatly improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.

FIG. 1 is a flow chart of a distributed computing method provided by an embodiment of the present disclosure;

fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present disclosure;

fig. 3 is a schematic diagram of an application scenario provided in an embodiment of the present disclosure;

fig. 4 is a schematic diagram of an application scenario provided in an embodiment of the present disclosure;

FIG. 5 is a basic architecture diagram of an overall system provided by another embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a distributed computing device provided by an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of an embodiment of an electronic device according to an embodiment of the disclosure.

Detailed Description

In order that the above objects, features and advantages of the present disclosure may be more clearly understood, a further description of aspects of the present disclosure will be provided below. It should be noted that, without conflict, the embodiments of the present disclosure and features in the embodiments may be combined with each other.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the disclosure.

It should be noted that, the user code (including but not limited to the code written by the user on the user interface, or the code written by the user through the software development tool, etc.) and the data (including but not limited to the data used for analysis, the stored data, the displayed data, etc.) related to the present application are both information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and are provided with corresponding operation entries for the user to select authorization or rejection.

In addition, the distributed computing method provided by the application can involve the following terms for explanation, and the details are as follows:

ECI: the elastic container instance (Elastic Container Instance) is a agile secure Serverless container running service. The container can be run by only providing the packaged mirror image without managing the bottom server.

Serverless, also called Serverless. Serverless emphasizes an architecture idea and service model that allows developers to focus on application business logic without concern for infrastructure (servers, etc.). Serverless is also a next generation computing engine.

STAC: the spatiotemporal asset catalog (Spatio Temporal Asset Catalog, star) series specification aims to standardize the way in which geospatial asset metadata is structured and queried, being any file of earth information representing a certain place and time.

Distributed type: a distributed system is a system consisting of a set of computer nodes that communicate over a network to coordinate work in order to accomplish a common task.

Spark: is a memory-based, general and extensible distributed big data computing engine framework.

Remote sensing: refers to non-contact and long-distance detection technology, and generally refers to detection of radiation and reflection characteristics of electromagnetic waves of an object by using a sensor and a remote sensor.

RDD: is a resilient distributed data set, RDD is an acronym for Resilient Distributed Dataset.

Kyuubi: is a distributed and multi-tenant gateway, and provides SQL and code query services for a data lake query engine (such as Spark).

Data lake: is where data is centrally stored, such as data storage space.

In the remote sensing satellite industry, how to rapidly process and analyze a large amount of remote sensing data is a problem to be solved. In addition, not only is a large amount of remote sensing data generated in the remote sensing satellite industry, but also a large amount of data may be generated in other scenes, for example, a large amount of sensing data is generated in the drive test process of an automatic driving vehicle. However, the calculation process of the large amount of data is slow due to the large calculation amount of the large amount of data, and the efficiency of processing and analyzing the large amount of data is reduced.

In view of this problem, embodiments of the present disclosure provide a distributed computing method, which is described below in connection with particular embodiments.

Fig. 1 is a flowchart of a distributed computing method according to an embodiment of the present disclosure. The method may be performed by a distributed computing device, which may be implemented in software and/or hardware, which may be configured in an electronic device, such as a server. The server may be a cloud server, for example, the cloud may be deployed with a plurality of cloud servers, where each cloud server has processing resources such as computation and storage. At the cloud, a service may be provided by multiple cloud servers, although one cloud server may provide one or more services. The cloud may provide the service by providing a service interface to the outside, and the user invokes the service interface to use the corresponding service. The service interface includes a software development kit (Software Development Kit, abbreviated as SDK), an application program interface (Application Programming Interface, abbreviated as API), and the like. Therefore, the distributed computing method provided in this embodiment may be performed by a plurality of cloud servers in the cloud, or may be performed by one cloud server. Assume that a cloud server performs a distributed computing method, and the cloud server may specifically be the server 22 shown in fig. 2. In addition, the distributed computing method described in this embodiment may be applied to an application scenario as shown in fig. 2. As shown in fig. 2, the application scenario includes a terminal 21 and a server 22, where the terminal 21 specifically includes a mobile phone, a computer, a tablet computer, or the like. The following describes the method in detail with reference to fig. 2, and as shown in fig. 1, the method specifically includes the following steps:

S101, acquiring codes written by a user on a user interface, wherein the codes are used for describing calculation logic aiming at target data.

For example, the terminal 21 shown in fig. 2 may be in particular a user terminal, the terminal 21 may be provided with a user interface on which a user may write a code. The code describes the calculation logic for the target data according to the user's business needs. The target data may be a large amount of data such as remote sensing data and sensing data as described above. Further, the terminal 21 may transmit a code written by the user on the user interface to the server 22.

S102, analyzing the codes into a grammar tree, and generating one or more logic execution plans according to the grammar tree.

When the server 22 receives the user-written code, the code may be parsed into a syntax tree.

Optionally, parsing the code into a syntax tree includes: abstracting the input, output and operation of each part in the code; and arranging the input, output and operation of each part into the grammar tree by using a preset data structure.

For example, the server 22 may abstract the inputs, outputs, and operations of each portion of the code and then reschedule the inputs, outputs, and operations of each portion of the code into a syntax tree, e.g., an abstract syntax tree (abstract syntax code, AST), using a preset data structure, e.g., a java script object profile (Java Script Object Notation, JSON).

Further, server 22 may generate one or more logical execution plans from the syntax tree. It will be appreciated that the logical execution plan is a mapping of the syntax tree, and that some optimization operations may be performed when generating the logical execution plan, and thus the logical execution plan may be a subset of the syntax tree.

S103, connecting each operator according to the dependency relationship among each operator in the logic execution plans aiming at each logic execution plan in the one or more logic execution plans to form a calculation link corresponding to the logic execution plans, and dividing the calculation link into a plurality of calculation tasks.

Since the syntax tree includes an operation for each part of the code, each logical execution plan includes at least a part of the operation in the code, each operation can be regarded as a function, and a function can be regarded as an operator, and thus each logical execution plan can include a plurality of operators, respectively. Because of the certain front-back dependency relationship among the operations in the code, the certain front-back dependency relationship among the operators in each logic execution plan exists. Therefore, according to the front-back dependency relationship between the operators in each logic execution plan, the operators in each logic execution plan can be respectively connected to form a computation link (Directed Acyclic Graph, DAG) corresponding to each logic execution plan. Further, the computing link may be divided into a plurality of computing tasks, each of which is an execution task with a task (task) as a minimum computing unit.

And S104, distributing the plurality of calculation tasks to a plurality of distributed calculation nodes for parallel execution, and obtaining a calculation result aiming at the target data.

Further, the server 22 may distribute the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution. For example, server 22 divides the syntax tree into two logical execution plans, and since each logical execution plan may correspond to one computing link, server 22 may generate two computing links. Further, the server 22 may divide each computing link into a plurality of computing tasks, e.g., one computing link into a plurality of computing tasks and another computing link into a plurality of computing tasks. Further, the server 22 may distribute the plurality of computing tasks into which one of the computing links is divided to a plurality of distributed computing nodes, which may be, for example, the computing node 31, the computing nodes 32, …, the computing node 33 as shown in fig. 3, to be executed in parallel. In addition, the server 22 may also distribute the multiple computing tasks into which another computing link is divided to multiple distributed computing nodes, such as the computing node 34, the computing nodes 35, …, and the computing node 36 shown in fig. 3, for parallel execution. Specifically, for each of the computing nodes 31, 32, …, 33, 34, 35, …, and 36, at least one computing task may be executed on the computing node, after the computing node executes the at least one computing task, a computing result of the at least one computing task may be returned to the server 22, and the server 22 may determine a computing result of the target data according to the computing results returned by the computing nodes 31, 32, …, 33, 34, 35, …, and 36, respectively.

In addition, if the server 22 divides the syntax tree into a logical execution plan, the server 22 may generate a calculation link since the logical execution plan may correspond to a calculation link. Further, the server 22 may divide the computing link into a plurality of computing tasks and distribute the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution, for example, the computing nodes 41, 42, …, 43 as shown in fig. 4. At least one calculation task can be executed on each of the calculation nodes 41, 42, … and 43, after the calculation node executes the at least one calculation task, the calculation result of the at least one calculation task can be returned to the server 22, and the server 22 can determine the calculation result of the target data according to the calculation results returned by the calculation nodes 41, 42, … and 43 respectively.

Embodiments of the present disclosure generate one or more logical execution plans from a syntax tree by retrieving code written by a user on a user interface and parsing the code into the syntax tree. And then, connecting each operator according to the dependency relationship among each operator in the logic execution plan aiming at each logic execution plan in one or more logic execution plans to form a calculation link corresponding to the logic execution plan, dividing the calculation link into a plurality of calculation tasks, distributing the plurality of calculation tasks to a plurality of distributed calculation nodes, and executing the calculation tasks in parallel to obtain a calculation result aiming at the target data. When the target data is a large amount of data, for example, a large amount of remote sensing data, the processing and analysis processes of the large amount of remote sensing data can be distributed to a plurality of distributed computing nodes to be executed in parallel by the method of the embodiment, so that the efficiency of processing and analyzing the large amount of remote sensing data is greatly improved.

The distributed computing method according to this embodiment will be described in detail below with reference to fig. 5. A basic architecture diagram of the overall system is shown in fig. 5. The system mainly comprises three parts, a first part is a part from a user to a second load balancing, a second part is a part from the second load balancing to a data release service, and a third part is a right part as shown in fig. 5. The first part is that the access layer is requested, for example, the user performs code editing on a web page that can provide an interactive web page for the user terminal by calling a Zhu Pite editor (jupyter notebook) running in the server cluster. Further, a software development kit (Software Development Kit, SDK) Engine (Engine) Service (Service) parses the code and converts the code into a logical execution plan. The code is used to describe the computing logic for the target data, and the corresponding computing types are classified into online computing (Interactive) and offline computing (Batch). The second part is an operator chained call layer, analyzes the input logic execution plan, generates a physical execution plan which can be executed according to the logic execution plan, and outputs the executed calculation result in different modes according to different calculation types. The third part is a resource scheduling layer, before the arrival of a computing request, the initialization of the environment and the computing resource allocation are needed, resources are applied to a container cluster management system (Kubernetes, K8S for short) through configuration, and when sufficient resources exist, a pod is started so as to pull up a computing node. Where in Kubernetes clusters Pod is the basis for all traffic types, also the minimum unit level of K8S management, which is a combination of one or more containers. These containers share specifications of storage, networks, and namespaces, and how to operate. In Pod, all containers are arranged and scheduled identically and run in a shared context. For a particular application, pod are their logical hosts, pod contains a plurality of application containers related to traffic.

Specifically, in the embodiment shown in fig. 5, the distributed computing method specifically includes the following steps:

step 1: the code is converted into a syntax tree.

For example, the user writes code on an interactive web page provided by jupyter notbook, and the present embodiment is not limited to the programming language used by the user when writing code, and may be, for example, a computer programming language such as python (python). The code describes the computational logic of target data, such as telemetry or other sensory data, based on user traffic demand. Further, SDK Engine Service converts the code into a syntax tree. Specifically, the input, output and operation of each part in the code are abstracted, and then the input, output and operation are rearranged into a grammar tree by using a JSON data structure, and the grammar tree can be recorded as a function Json. Specifically, the grammar tree contains all calculation logic and calculation nodes required by codes, so that optimization and execution of subsequent iterative calculation are facilitated. Wherein SDK Engine Service may be deployed in one or more cloud servers in the cloud or SDK Engine Service may be deployed on one or more computers of a cluster.

Step 2: the user request is distributed according to the calculation type.

For example, overall computation falls into two broad categories: on-line computing and off-line computing, wherein on-line computing generally requires higher demands on low latency, and for on-line computing, it is necessary to split target data to be processed into a plurality of sub-data, then compute the plurality of sub-data in parallel, and respond to the computation result to a user quickly according to the parallel computation result of the plurality of sub-data. While offline computation is a full-scale, oversized dataset computation with higher resolution accuracy and longer time consumption.

In one possible implementation, generating one or more logical execution plans from the syntax tree includes: if the calculation type aiming at the target data is online calculation, splitting the target data into a plurality of sub-data; and generating a logic execution plan corresponding to each piece of sub data respectively according to the position of each piece of sub data in the target data.

For example, the interactive web page includes not only a code editing area, but also control buttons, such as a submit button. After the user edits the completion code on the interactive web page, the submit button may be clicked, at which point SDK Engine Service may receive a user request including the code therein, and further, SDK Engine Service converts the code into a grammar tree.

For online computing, SDK Engine Service may send the syntax tree to a geographic server (GeoServer) that splits the target data to be processed by the code into multiple sub-data, e.g., the target data may be an image that is stored in an object store server (Object Storage Service), which may split the image into multiple tiles, each tile size 256x256, each tile may be denoted as one sub-data. For example, the geographic server splits the image into N tiles. Further, the geographic server may determine the location information, i.e., boundary information, of each tile in the N tiles in the image, and generate a logic execution plan corresponding to the tile according to the location information of each tile in the image and the syntax tree, that is, split the syntax tree into N logic execution plans according to the location information of each tile in the N tiles in the image. In addition, the geographic server may also generate N computing requests, where each computing request in the N computing requests includes location information of a tile and a logical execution plan corresponding to the tile. The geographic server may then distribute the N computing requests to the stark (Spark) gateways through load balancing (Server Load Balance), each Spark gateway may receive one or more computing requests. Spark is an open source cluster computing framework whose primary purpose is to process data generated in real time.

For offline computation, the syntax tree may be sent SDK Engine Service to a Spark gateway through load balancing without splitting of tiles. Generating one or more logical execution plans from the syntax tree, comprising: and if the calculation type aiming at the target data is offline calculation, generating the logic execution plan according to the grammar tree. For example, in one possible implementation, the Spark gateway may generate a logical execution plan from the syntax tree. Alternatively, SDK Engine Service may generate a logical execution plan from the syntax tree and then send the logical execution plan to a Spark gateway.

Step 3: selecting a computing node, managing and routing a computing request.

For example, for online computing, after N computing requests are distributed to multiple Spark gateways, each Spark gateway uniformly routes and schedules one or more computing requests it receives. For example, each Spark gateway may perform a permission check on each computing request it receives, pick a server-less (serverless) for each computing request, and forward the computing request to the server-less. Specifically, a server may receive one or more computing requests. In this embodiment, multiple geographic servers may form a cluster, multiple Spark gateways may form a cluster, and multiple servers may form a cluster, which may be different clusters or the same cluster. The nodes in each cluster can be laterally expanded, for example, capacity expansion or capacity contraction is performed for different request orders, that is, the nodes are increased or decreased for different request orders. Wherein, spark gateway and server can interact. Specifically, the Spark gateway may trigger the server to manage the start and stop of the Spark computing instance and the passive cancellation of the computing request.

For offline computation, the Spark gateway may generate a logical execution plan from the syntax tree. Alternatively, when the Spark gateway receives the logic execution plan sent by SDK Engine Service, the logic execution plan may be forwarded to the serverless.

Step 4: operator chained calls.

For example, each server may also correspond to a Client, which may specifically be a K8S Client (Client). The K8S client may interact with the K8S cluster. Specifically, the K8S cluster may include a main process (K8S Master), a Spark Driver process (Spark Driver Pod), and an execution process (Executor Pod) as shown in fig. 5. The K8S Master comprises a process Scheduler (Pod Scheduler) and an application program interface Server (API Server). The K8S Master is used for applying the resources required by the Spark Driver Pod, and after the K8S Master applies for the resources to the Spark Driver Pod, the K8S Master can schedule the Spark Driver Pod, namely, start the Spark Driver Pod. Specifically, the K8S Master and Spark Driver Pod may run on different servers in the K8S cluster, respectively, or the K8S Master and Spark Driver Pod may run on the same server in the K8S cluster. When the Spark Driver Pod is started, the Spark Driver Pod can apply for resources required by the Executor Pod to the K8S Master, and when the K8S Master allocates corresponding resources to each Executor Pod, the K8S Master can schedule each Executor Pod, namely, start each Executor Pod. In addition, the K8S Master may also send a monitoring event for the execution process to the Spark Driver Pod, so that the Spark Driver Pod periodically acquires heartbeat messages of each exechamor Pod, so that the Spark Driver Pod determines whether each exechamor Pod survives. In addition, each Executor Pod may be registered in the Spark Driver Pod, so that the Spark Driver Pod may manage each Executor Pod, for example, start the Executor Pod, stop the Executor Pod.

For online computing, each serverless shown in fig. 5 may receive one or more computing requests respectively, and further, each serverless may send information to the K8S cluster through the K8S Client, where the information includes address information of the activated Spark Driver Pod, system parameters, and a logic execution plan included in each computing request. The system parameters may include a system parameter of the K8S Master, a system parameter of the Spark Driver Pod, and a system parameter of the Executor Pod. The system parameters of the K8S Master may specifically be a central processing unit (Central Processing Unit, CPU) and a memory occupied by the K8S Master. The system parameter of the Spark Driver Pod may specifically be a CPU, a memory, etc. occupied by the Spark Driver Pod. The system parameters of the Executor Pod can be CPU, memory and the like occupied by the Executor Pod.

It will be appreciated that for online computing, a server may choose one or more Spark Driver Pod that has been started. In one possible implementation, the number of calculation requests received by the server is consistent with the number of started Spark Driver Pod selected by the server. For example, when a computation request is received by the serverless, the serverless may select a Spark Driver Pod that is enabled. Further, the Spark Driver Pod may receive information, such as information sent by server, including address information of the Spark Driver Pod, system parameters, and a logic execution plan included in the calculation request. Further, the Spark Driver Pod can parse the logical execution plan. Then, according to the front-back dependency relationship between each operator in the logic execution plan, each operator in the logic execution plan is connected, for example, in series, so as to generate a calculation link. The core of the concatenation is a resilient distributed dataset of Spark RDDs, on which each function in the logical execution plan acts to form a completed computational link. In addition, in the tandem process, a great amount of optimization operations can be performed on the logic execution plan, for example, data filtering, searching and the like are pre-arranged before triggering calculation as much as possible through predicate pushing, and useless data reading is reduced; and deleting and combining repeated nodes in the logic execution plan through pruning and combining, so that the whole calculation link is simplified, and repeated calculation is avoided. In addition, in the tandem process, the real calculation process is not triggered, and only when a special operator is encountered, the real calculation process is triggered, namely, when the special operator is encountered, the conversion from the logic execution plan to the physical execution plan is performed, and the calculation link is divided into a plurality of calculation tasks. The server may post the plurality of computing tasks to the launched Spark Driver Pod. Further, the Spark Driver Pod submits a plurality of computing tasks to the computing node, and obtains a computing result, for example, the computing tasks are distributed to a plurality of Executor Pod managed by the Spark Driver Pod, so that each Executor Pod performs at least one computing task, that is, one Executor Pod can be regarded as one computing node. For another example, when the server receives multiple computing requests, the server may select an activated Spark Driver Pod for each computing request, so that each Spark Driver Pod processes multiple computing tasks corresponding to one computing request, and similarly, each Spark Driver Pod may allocate multiple computing tasks corresponding to one computing request to multiple Executor pods managed by the Spark Driver Pod, so that each Executor Pod executes at least one computing task. In yet another possible implementation, the number of calculation requests received by the server is not related to the number of started Spark Driver Pod selected by the server. For example, each started Spark Driver Pod selected by server processes at least a plurality of computing tasks corresponding to one computing request.

Optionally, distributing the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution, to obtain a computing result for the target data, including: distributing a plurality of calculation tasks corresponding to the logic execution plans to a plurality of distributed calculation nodes for parallel execution aiming at the logic execution plans corresponding to each piece of sub data respectively to obtain calculation results corresponding to each piece of sub data respectively; and generating a calculation result aiming at the target data according to the calculation result respectively corresponding to each piece of sub data.

For example, for online computing, when one or more computing requests are received by the serverless, each computing request includes location information for a tile, and a logical execution plan for the tile. Therefore, the server can select a started Spark Driver Pod for each calculation request received by the server, and deliver a plurality of calculation tasks corresponding to one calculation request to one started Spark Driver Pod, so that the Spark Driver Pod distributes the plurality of calculation tasks to a plurality of Executor pods, and each Executor Pod executes at least one calculation task to obtain a calculation result of each calculation task. In addition, the multiple Executor Pod under the Spark Driver Pod may further feed back the calculation results obtained by the multiple Executor Pod to the Spark Driver Pod, so that the Spark Driver Pod may generate the calculation result of the tile corresponding to the calculation request according to the calculation result of each calculation task in the multiple calculation tasks. Further, the Spark Driver Pod may feed back the calculation result of the tile to the server, where the server receives the calculation result of the tile fed back by each Spark Driver Pod selected by the server, and therefore, each server may receive the calculation result of at least one tile. Further, each server may sequentially transmit the calculation result of at least one tile received by each server upwards, and when the calculation result of each tile reaches the geographic server, the geographic server fuses the calculation results of each tile to obtain the calculation result of the target data, for example, an image.

For offline computation, when the server receives the logic execution plan, the server sends information to the K8S cluster through the K8S Client, where the information includes address information of the started Spark Driver Pod, system parameters, and the logic execution plan selected by the server. Further, spark Driver Pod receives the information and parses the logic execution plan. Then, according to the front-back dependency relationship between each operator in the logic execution plan, each operator in the logic execution plan is connected, for example, in series, so as to generate a calculation link. The Spark Driver Pod can divide the computing link into a plurality of computing tasks, and submit the computing tasks to the computing node to obtain a computing result of the target data.

Step 5: and (5) resource scheduling and telescoping.

For example, for online or offline computing, when the computation is triggered in fact, the DAG is divided into a plurality of computing tasks, and in order to run each computing task, a computing node, which may be an Executor Pod as described above, is applied to the K8S cluster, and each computing node performs at least one computing task, so that the plurality of computing tasks may be performed in parallel on the plurality of computing nodes. The plurality of computing nodes is denoted as a plurality of distributed computing nodes.

On the basis of the above embodiment, after the plurality of computing tasks are distributed to a plurality of distributed computing nodes and executed in parallel, the method further includes: if the number of the calculation tasks is greater than a preset threshold, increasing the number of the distributed calculation nodes; and if the time length of the distributed computing node in the idle state is greater than or equal to the preset time length, recovering the computing resources of the distributed computing node.

For example, when the number of computing tasks into which the DAG is divided is greater than a preset threshold, the system will automatically pull up the computing nodes matching the number for extended computation. If no computing task is needed at present, and the time length of the computing node in the idle state is greater than or equal to the preset time length, the computing node can be automatically paused, and computing resources of the computing node are recovered to the K8S cluster so that the K8S cluster can be used for the next distribution. Therefore, the computing resources can be fully utilized, the capacity is expanded when busy, and the capacity is reduced when idle, so that unnecessary resource waste is avoided.

Step 6: and (5) showing and persistence of the calculation result.

For online computing, the method further comprises: and displaying the calculation result of the target data on the user interface. For example, each serverless may receive the results of the computation of at least one tile. Serverless may compress and escape the calculation of each tile to form a template format that may be recognized by the geographic server. Further, each server may sequentially transmit the calculation result of at least one tile received by each server upwards, and when the calculation result of each tile reaches the geographic server, the geographic server fuses the calculation results of each tile to obtain the calculation result of the target data, for example, an image. Further, the geographic server sends the image's results of the computation to SDK Engine Service, SDK Engine Service for presentation in a user interface, such as an interactive web page where the user is programmed, for viewing by the user.

For offline computing, the computing tasks are distributed to a plurality of distributed computing nodes to be executed in parallel, and after a computing result aiming at the target data is obtained, the method further comprises: storing the calculation result of the target data in a distributed file form in an object storage server; and sending the distributed file to a user terminal based on a download request of the user for the distributed file. For example, the calculation result of the target data is stored in the OSS in the form of a distributed file, and a data release service is triggered subsequently based on a download request of the user for the distributed file, so that the distributed file is sent to the user terminal, and the user terminal may specifically be the user terminal displaying the interactive webpage. Specifically, the user terminal may display the download address of the distributed file in the interactive web page, and when the user clicks the download address, the OSS sends the distributed file to the user terminal, so that the user terminal downloads the distributed file locally.

In this embodiment, the distributed computing method may be performed by a plurality of cloud servers in the cloud, which may include SDK Engine Service, geographic servers, spark gateways, servers, K8S clusters, OSS, etc. as shown above.

According to the embodiment, the user writes the codes on the interactive webpage, so that the operation is simple and flexible, operators are easy to expand, and the use threshold of the user, such as a remote sensing professional, on remote sensing data is reduced. Because the cloud server has rich professional remote sensing function computing power, users can be combined at will so as to meet various complex demand scenes. In addition, after the ultra-large scale remote sensing data is divided into a plurality of tiles, the distributed tile calculation based on pyramid layering can be applied to RDD chain calculation rules among a plurality of operators aiming at each tile, so that the complex calculation capacity and the calculation efficiency are greatly improved. In addition, since the tile data is calculated in the memory, input/Output (IO) operations required by the intermediate result drop in the calculation process are reduced, and interaction between the traditional calculation and a disk and unnecessary network IO are avoided. In addition, by means of dynamic expansion and contraction of the computing nodes, tasks of different computing types can fully exert computing characteristics of the tasks, and dynamic adjustment of distributed computing is completed by means of resource expansion and contraction.

Fig. 6 is a schematic structural diagram of a distributed computing device according to an embodiment of the present disclosure. The distributed computing device provided by the embodiments of the present disclosure may execute the processing flow provided by the embodiments of the distributed computing method, as shown in fig. 6, the distributed computing device 60 includes:

An acquisition module 61 for acquiring code written by a user on a user interface, the code describing computing logic for target data;

a parsing module 62, configured to parse the code into a syntax tree;

a generating module 63, configured to generate one or more logic execution plans according to the syntax tree;

the connection module 64 is configured to, for each of the one or more logic execution plans, connect each operator according to a dependency relationship between the operators in the logic execution plan, form a computing link corresponding to the logic execution plan, and divide the computing link into a plurality of computing tasks;

and the distribution module 65 is configured to distribute the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution, so as to obtain a computing result for the target data.

Optionally, the distributed computing device 60 further includes: an add module 66 and a recycle module 67; after the distribution module 65 distributes the plurality of computing tasks to the plurality of distributed computing nodes for parallel execution, if the number of the computing tasks is greater than a preset threshold, the increasing module 66 is configured to increase the number of the distributed computing nodes; if the time length of the distributed computing node in the idle state is greater than or equal to the preset time length, the reclamation module 67 is configured to reclaim the computing resources of the distributed computing node.

Optionally, when the parsing module 62 parses the code into a syntax tree, the parsing module is specifically configured to:

abstracting the input, output and operation of each part in the code;

and arranging the input, output and operation of each part into the grammar tree by using a preset data structure.

Optionally, when the generating module 63 generates one or more logic execution plans according to the syntax tree, the generating module is specifically configured to:

if the calculation type aiming at the target data is online calculation, splitting the target data into a plurality of sub-data;

and generating a logic execution plan corresponding to each piece of sub data respectively according to the position of each piece of sub data in the target data.

Optionally, when the distribution module 65 distributes the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution to obtain a computing result for the target data, the method is specifically used for:

distributing a plurality of calculation tasks corresponding to the logic execution plans to a plurality of distributed calculation nodes for parallel execution aiming at the logic execution plans corresponding to each piece of sub data respectively to obtain calculation results corresponding to each piece of sub data respectively;

and generating a calculation result aiming at the target data according to the calculation result respectively corresponding to each piece of sub data.

Optionally, the distributed computing device 60 further includes: and a display module 68 for displaying the calculation result of the target data on the user interface.

Optionally, when the generating module 63 generates one or more logic execution plans according to the syntax tree, the generating module is specifically configured to: and if the calculation type aiming at the target data is offline calculation, generating the logic execution plan according to the grammar tree.

Optionally, the distributed computing device 60 further includes: a storage module 69 and a transmission module 610. After the distribution module 65 distributes the multiple computing tasks to multiple distributed computing nodes to execute in parallel, and obtains a computing result for the target data, the storage module 69 is configured to store the computing result for the target data in a form of a distributed file in an object storage server; the sending module 610 is configured to send the distributed file to a user terminal based on a download request of the user for the distributed file.

The distributed computing device of the embodiment shown in fig. 6 may be used to implement the technical solution of the above-described method embodiment, and its implementation principle and technical effects are similar, and are not described herein again.

The foregoing describes the internal functions and structures of a distributed computing device that may be implemented as an electronic device. Fig. 7 is a schematic structural diagram of an embodiment of an electronic device according to an embodiment of the disclosure. As shown in fig. 7, the electronic device includes a memory 71 and a processor 72.

The memory 71 is used to store programs. In addition to the programs described above, the memory 71 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.

The memory 71 may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The processor 72 is coupled to the memory 71, and executes programs stored in the memory 71 for:

Further, as shown in fig. 7, the electronic device may further include: communication component 73, power component 74, audio component 75, display 76, and the like. Only some of the components are schematically shown in fig. 7, which does not mean that the electronic device only comprises the components shown in fig. 7.

The communication component 73 is configured to facilitate communication between the electronic device and other devices, either wired or wireless. The electronic device may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 73 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 73 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

A power supply assembly 74 provides power to the various components of the electronic device. The power components 74 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for electronic devices.

The audio component 75 is configured to output and/or input audio signals. For example, the audio component 75 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 71 or transmitted via the communication component 73. In some embodiments, the audio component 75 further comprises a speaker for outputting audio signals.

The display 76 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation.

In addition, the embodiment of the present disclosure also provides a computer-readable storage medium having stored thereon a computer program that is executed by a processor to implement the distributed computing method described in the above embodiment.

It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown and described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed computing method, wherein the method comprises:

2. The method of claim 1, wherein after distributing the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution, the method further comprises:

if the number of the calculation tasks is greater than a preset threshold, increasing the number of the distributed calculation nodes;

and if the time length of the distributed computing node in the idle state is greater than or equal to the preset time length, recovering the computing resources of the distributed computing node.

3. The method of claim 1, wherein parsing the code into a syntax tree comprises:

abstracting the input, output and operation of each part in the code;

4. The method of claim 1, wherein generating one or more logical execution plans from the syntax tree comprises:

5. The method of claim 4, wherein distributing the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution results in a computation result for the target data, comprising:

6. The method of claim 5, wherein the method further comprises:

and displaying the calculation result of the target data on the user interface.

7. The method of claim 1, wherein generating one or more logical execution plans from the syntax tree comprises:

and if the calculation type aiming at the target data is offline calculation, generating the logic execution plan according to the grammar tree.

8. The method of claim 7, wherein distributing the plurality of computing tasks to a plurality of distributed computing nodes for parallel execution, resulting in a computing result for the target data, the method further comprising:

storing the calculation result of the target data in a distributed file form in an object storage server;

and sending the distributed file to a user terminal based on a download request of the user for the distributed file.

9. A distributed computing device, comprising:

The analysis module is used for analyzing the codes into grammar trees;

10. An electronic device, comprising:

a memory;

a processor; and

a computer program;

wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the method of any of claims 1-8.