CN114691233A - Remote sensing data processing plug-in distributed scheduling method based on workflow engine - Google Patents
Remote sensing data processing plug-in distributed scheduling method based on workflow engine Download PDFInfo
- Publication number
- CN114691233A CN114691233A CN202210256586.2A CN202210256586A CN114691233A CN 114691233 A CN114691233 A CN 114691233A CN 202210256586 A CN202210256586 A CN 202210256586A CN 114691233 A CN114691233 A CN 114691233A
- Authority
- CN
- China
- Prior art keywords
- plug
- service
- ins
- scheduling
- slave node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 87
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000008569 process Effects 0.000 claims abstract description 40
- 230000015654 memory Effects 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 9
- 230000000007 visual effect Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 238000009826 distribution Methods 0.000 claims description 4
- 238000012986 modification Methods 0.000 claims description 4
- 230000004048 modification Effects 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000011161 development Methods 0.000 abstract description 3
- 238000003860 storage Methods 0.000 description 8
- 238000012937 correction Methods 0.000 description 7
- 238000004519 manufacturing process Methods 0.000 description 6
- 230000006837 decompression Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 230000005855 radiation Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
- G06F9/44526—Plug-ins; Add-ons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a distributed scheduling method for remote sensing data processing plug-ins based on a workflow engine. The logic control service carries out logic scheduling management on the execution process, arranges the whole operation flow, configures the operation parameters and initiates the whole flow; the main node service is responsible for business plug-in management and business scheduling execution, receives parameter information to be executed transmitted by the logic control service, collects execution result information and feeds the execution result information back to the logic control service, coordinates each computing resource in the cluster and executes a load balancing strategy; the slave node service is responsible for receiving the execution task parameters sent by the master node service, calling the relevant plug-in to execute, and reporting the execution result and the use condition of the slave node resource to the master node service. The invention fully utilizes the flexible scheduling capability of the workflow technology, simplifies the development process, provides the WYSIWYG flow arrangement capability and realizes the distributed parallel scheduling capability of the complex remote sensing data processing.
Description
Technical Field
The invention belongs to the technical field of remote sensing data processing, and particularly relates to a distributed scheduling method for remote sensing data processing plug-ins based on a workflow engine.
Background
At present, the number of the emitted satellites is continuously increased, satellite image data processing plug-ins are continuously increased, a matched satellite data processing system needs to adapt to the data processing and product production capacity of various satellites, a new technical framework needs to be utilized facing the production and subsequent application requirements of remote sensing data from original code streams to various levels of data products, a distributed scheduling method is designed, the reasonable scheduling of various processing plug-ins under a cluster resource environment is completed, and the automatic production and processing of the remote sensing data are realized.
The workflow technology is used as a calculation model of a workflow, logic and rules of how work in the workflow is organized front and back are expressed in a computer by a proper model and are calculated, and currently, mainstream workflow engines include Activiti, OSWorkflow, JBPM and the like.
Disclosure of Invention
In order to solve the problems, the invention introduces an open-source workflow engine, provides a distributed scheduling method of remote sensing data processing plug-ins based on the workflow engine, and designs a logic control service and master-slave node two-stage scheduling mechanism based on the workflow engine based on a micro-service idea.
In order to achieve the purpose, the invention adopts the technical scheme that:
a remote sensing data processing plug-in distributed scheduling method based on a workflow engine comprises the following steps:
(1) carrying out logic scheduling management on the execution process through a logic control service, arranging the whole operation flow, configuring the operation parameters and initiating the whole flow;
performing service plug-in management and service scheduling through the master node service, receiving parameter information to be executed transmitted by the logic control service, collecting execution result information and feeding back the execution result information to the logic control service, coordinating all computing resources in a cluster, executing a load balancing strategy and ensuring reasonable distribution of tasks;
receiving an execution task parameter sent by the master node service through the slave node service, calling and executing a relevant plug-in, and reporting an execution result and the use condition of slave node resources to the master node service;
(2) for the remote sensing data processing plug-in, the relevant information of the plug-in is described according to the unified template, the host node service catalogs and stores the plug-in, manages the plug-in a visual mode and provides functions of adding, deleting, modifying and searching the plug-in;
(3) the logic control service reads and processes a plug-in database table based on a workflow engine, analyzes the warehoused processing plug-in information, organizes and manages various processing plug-ins in a visual mode, and finishes the process arrangement and parameter transmission setting of the processing plug-ins in a dragging mode;
(4) after the process arrangement is finished, executing the process scheduling according to the process input parameters, wherein in the executing process, the logic control service determines task plug-ins needing to be executed next step according to the process arrangement result and transmits the parameters to the master node service, and the master node service generates calling parameters and sends the calling parameters to the slave node service needing to be scheduled according to the deployment condition of the processing plug-ins and the service condition of each slave node resource based on a polling scheduling strategy;
(5) and the slave node service generates a specific plug-in calling parameter according to the parameter information sent by the master node service, and completes the calling of the plug-ins.
Furthermore, in the step (1), the logic control service realizes a plurality of execution strategies of merging, splitting, parallel, asynchronous and parameter pairing through the expansion and the transformation of the workflow engine, and supports any combination of the execution strategies; and aiming at the same plug-in processing step in the processing process, service instantiation segmentation is carried out, so that the same plug-in can generate a plurality of service instances, and the instances are independent and can be processed in parallel.
Further, in the step (2), resource information occupied by various plug-ins during execution is determined according to the types of the business plug-ins, including the number of cores of the CPU and the memory capacity, and the relevant information of the plug-ins includes the name of the plug-ins, the function description of the plug-ins, the number of occupied CPUs, the number of occupied memories, and the calling service interfaces of the plug-ins.
Further, in step (4), it is first determined in which slave nodes the to-be-called plug-ins are deployed, the master node service performs calculation according to the current resource usage reported by the slave node services, that is, the number of idle CPUs and the memory capacity of the slave nodes, in combination with the resource information occupied by the to-be-called processing plug-ins when executed, and finally determines which node to call, and the calculation formula is as follows:
C= Cn -Ca
M= Mn-Ma
cn is the total core number of the current polling slave node CPU, Ca is the core number occupied when the processing plug-in to be called is executed, Mn is the total memory amount of the slave node, and Ma is the memory capacity occupied when the processing plug-in to be called is executed; and when both M and C are larger than 0, determining to call the processing plug-in of the current slave node.
Compared with the prior art, the invention has the advantages that:
1. the invention fully utilizes the flexible scheduling capability of the workflow technology, simplifies the development process, provides the WYSIWYG flow arrangement capability and realizes the distributed parallel scheduling capability of the complex remote sensing data processing.
2. The invention designs a logic control service and master-slave node two-stage scheduling mechanism based on a workflow engine, realizes the control of the whole processing flow by a multi-stage gradual progressive scheduling mode, and realizes each stage of scheduling module based on a micro-service idea, thereby improving the scheduling flexibility.
3. The invention fully combines the characteristics of the remote sensing data processing plug-in, and makes a plug-in load balancing scheduling strategy.
Drawings
FIG. 1 is a schematic diagram of the method of the present invention.
FIG. 2 is a diagram of the deployment relationship of master-slave node modules in a cluster environment according to the method of the present invention.
FIG. 3 is a flow chart of remote sensing image data from original code stream data receiving, product production to image intelligent processing.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
A distributed scheduling method for remote sensing data processing plug-ins based on a workflow engine specifically comprises the following steps:
(1) the logic control service carries out logic scheduling management on the execution process, arranges the whole operation flow, and is responsible for configuring the operation parameters and initiating the whole flow. The main node service is responsible for service plug-in management and service scheduling execution, receives parameter information to be executed transmitted by the logic control service, collects execution result information and feeds the execution result information back to the logic control service, and meanwhile coordinates each computing resource in the cluster, executes a load balancing strategy and ensures reasonable distribution of tasks. The slave node service is responsible for receiving an execution task parameter sent by the master node service, calling a relevant plug-in to execute, and reporting an execution result and the use condition of slave node resources to the master node service;
(2) for the remote sensing data processing plug-in, the relevant information of the plug-in is described according to the unified template, the host node service catalogs and stores the plug-in, manages the plug-in a visual mode and provides a plug-in addition, deletion, modification and check function;
(3) the logic control service is modified based on a workflow engine, reads a processing plug-in database table, analyzes the information of the processing plug-ins which are put in storage, visually organizes and manages various processing plug-ins, and finishes the process arrangement and parameter transmission setting of the processing plug-ins in a dragging mode;
(4) after the process arrangement is finished, scheduling and executing the process according to the process input parameters, wherein in the executing process, the logic control service determines task plug-ins needing to be executed next step according to the process arrangement result and transmits the parameters to the master node service, and the master node service generates calling parameters and sends the calling parameters to the slave node service needing to be scheduled according to the deployment condition of the processing plug-ins and the service condition of each slave node resource based on a polling scheduling strategy;
(5) and the slave node service generates a specific plug-in calling parameter according to the parameter information sent by the master node service, and completes the calling of the plug-ins.
In the step (1), the logic control service realizes multiple execution strategies of merging, splitting, paralleling, asynchronizing and parameter matching through the expansion and the transformation of the workflow engine, and supports any combination of the execution strategies. Service instantiation segmentation is carried out aiming at the same plug-in processing step in the processing process, so that the same plug-in can generate a plurality of service instances, and the instances are independent from each other and can be processed in parallel.
In the step (2), resource information occupied by various plug-ins during execution is determined according to the types of the business plug-ins, wherein the resource information comprises CPU core number and memory capacity, and the plug-in description information comprises information such as plug-in names, plug-in function description, CPU occupation quantity, memory occupation quantity, plug-in calling service interfaces and the like.
In step (4), it is first determined in which slave nodes the plug-in to be invoked is deployed, the master node service performs calculation according to the current resource usage reported by the slave node services, that is, the number of idle CPUs and the memory capacity of the slave nodes, in combination with the resource information occupied by the plug-in to be invoked when executing, and finally determines which node to invoke, the calculation formula is as follows:
C= Cn -Ca
M= Mn-Ma
cn is the total core number of the current polling slave node CPU, Ca is the core number occupied when the processing plug-in to be called is executed, Mn is the total memory amount of the slave node, Ma is the memory capacity occupied when the processing plug-in to be called is executed, and when M and C are both larger than 0, the processing plug-in of the current slave node is determined to be called.
The following is a more specific example:
a remote sensing data processing plug-in distributed scheduling method based on a workflow engine is characterized in that an open source workflow engine is introduced, a remote sensing data processing plug-in distributed scheduling method based on the workflow engine is provided, and a logic control service + master-slave node two-stage scheduling mechanism based on the workflow engine is designed based on a micro-service idea. The general principle framework of the method is shown in fig. 1, and mainly comprises three parts, namely logic control service, master node service and slave node service.
The logic control service performs logic scheduling management on the execution process, arranges the whole operation flow and is responsible for configuring the operation parameters. The main node service is responsible for service plug-in management and service scheduling execution, receives parameter information to be executed transmitted by the logic control service, collects execution result information and feeds the execution result information back to the logic control service, and meanwhile manages cluster resources, coordinates each computing resource in a cluster, executes a load balancing strategy and ensures reasonable distribution of tasks. The slave node service is responsible for receiving the execution task parameters sent by the master node service, calling the relevant plug-in to execute, and reporting the execution result and the use condition of the slave node resource to the master node service.
The method is suitable for calling application in a distributed cluster environment, under the cluster environment, computing resources are divided into a main node cluster and a slave node cluster, the main node cluster deploys a logic control service and a main node service, the slave node cluster deploys a slave node service and various service plug-ins, data interaction exists between the main node cluster and the slave node cluster and distributed shared storage, remote sensing data processing services are combined, and module deployment relation in the cluster environment is shown in figure 2.
And configuring plug-in description information with a fixed format according to the characteristics of various remote sensing data processing plug-ins, wherein the plug-in description information comprises plug-in names, plug-in function description, the number of occupied CPUs (central processing units), the number of occupied memories and the like. The number of occupied CPUs and the number of occupied memories are resource peaks occupied by the plug-in during execution, and when the relevant plug-in runs, the CPU resources and the memory resources are required to meet the requirements of the description information.
In general, a master node cluster needs 2 computing resources to form a master node and a slave node, and each computing resource deploys a logic control service and a master node service. And the host node service finishes cataloging and warehousing storage of the plug-ins according to the plug-in description information, performs plug-in management in a visual mode and provides visual plug-in addition, deletion, modification and check functions. The logic control service is transformed based on an open-source activiti workflow engine, the remote sensing data processing plug-ins which are put in storage are analyzed by reading database information, various remote sensing data processing plug-ins are visually organized and managed to form a plug-in arranging warehouse, visual parameter transmission and configuration functions are designed, and the flow arrangement and parameter transmission setting of the remote sensing data processing plug-ins are completed in a dragging mode. After the process arrangement is completed, scheduling execution is performed on the process according to input parameters by utilizing the scheduling capability of a workflow engine, in the execution process, a master node service determines a processing plug-in needed to be called next according to the process arrangement result, polling scheduling is performed on each slave node, scheduling parameters are generated according to the deployment condition of the processing plug-ins and the resource use condition (the idle CPU core number and the idle memory amount) of each slave node, and the scheduling parameters are sent to the slave node service needing to be scheduled, wherein the scheduling parameter information comprises key information such as the name of the plug-in needing to be executed, the input and output of the plug-in, the number of CPUs (central processing unit) and the memory amount needed when the plug-in is executed.
And deploying slave node services and various service plug-ins by the slave node cluster, determining the number of computing resources in the cluster according to the cluster scale and the service processing condition, and deploying different plug-ins by each slave node according to the respective resource configuration condition. As shown in fig. 2, plug-ins such as decryption decompression, formatting, etc. are deployed in the slave node 1 and the slave node 2, and such plug-ins need to occupy more memory resources and have relatively less requirements on CPU resources; the slave node 3 is provided with plug-ins such as geometric correction, radiation correction, image orthographic projection and the like, the processing objects of the plug-ins are mostly single-scene remote sensing data, parallel processing is involved, the requirement on the number of CPU cores is high, and the requirement on memory resources is relatively low; plug-ins such as target detection, target tracking, ground feature classification and the like are deployed in the slave nodes 4, and the plug-ins are performed based on a deep learning algorithm and need GPU resources. The decryption decompression plug-in completes the processing of the original code stream data, and the processing object is one track of original code stream data; the formatted scene plug-in completes the processing of the decrypted and decompressed data, and the processing objects are a plurality of decompressed files; and plug-in processing objects such as radiation correction, geometric correction, image orthography, target detection, ground object classification and the like are single-scene remote sensing image data.
The slave node service generates specific plug-in execution parameters locally according to the plug-in calling parameter information sent by the master node service, completes scheduling execution of the plug-ins, and sends execution result information to the master node service after processing is completed to inform the master node of success or failure of service execution.
The original code stream data and the file data generated in the execution process are both stored in the distributed shared storage, and the distributed shared storage performs data interaction with the master node cluster and the slave node cluster through a gigabit network.
Fig. 3 is a whole process from an original code stream to a level 2 product to image intelligent processing of remote sensing image data, and the whole scheduling process is explained with reference to fig. 3 as follows:
when the processing plug-in is called from the node, necessary operation parameters need to be transmitted, the method specially designs an XML format document for transmitting the operation parameters of all the professional processing plug-ins, standardizes the label meaning in the document, can carry out label modification and hierarchical expression aiming at the operation parameter requirements of different plug-ins by utilizing the characteristics of an extensible markup language, accurately transmits the complex operation parameter information to the processing plug-in, and ensures the stability of the operation of the plug-ins and the accuracy of the processing result. Taking decryption decompression and radiation correction operation parameter transfer documents as examples, specific contents of the formulated XML file are as follows:
after acquiring one track of original code stream data, the logic control service analyzes the parameter information of the original code stream, defines the service flow to be executed, generates the parameter information to be executed and sends the parameter information to the master node service, the master node service determines the slave node service to be scheduled according to the load balancing strategy and sends the parameter information to be executed to the slave node service, the slave node service generates a parameter document and calls a decryption decompression plug-in of the current slave node to process the data file of the original code stream, a plurality of files to be divided are generated according to the number of sensors, the generated data files are stored on a distributed shared storage and the execution result is fed back to the master node service, after the master node service receives the feedback information, a formatted scene example is respectively generated for each file to be divided according to the number of the files to be divided, and the master node service takes each example as a scheduling unit, and determining slave nodes to be used by each instance according to a load balancing strategy, wherein each slave node service respectively calls the locally formatted scene plug-in to complete processing, the generated scene data files are all stored in the distributed shared storage, and the execution result is fed back to the master node service to continuously complete the scheduling execution of the subsequent plug-ins. And the formatted scene instances are mutually independent and are processed in parallel, subsequent product production and intelligent processing instances are respectively established based on the scene data files generated by the instances, and the calling of the specific plug-in is completed according to a load balancing strategy. The whole process is finally completed through continuous iteration of the process, and the production from the satellite original data to the data products and the intelligent processing products is realized.
The method is realized through JAVA and JS languages, wherein the front-end interface is realized through JS language, and the rear end is realized through JAVA language. Meanwhile, decryption and decompression, formatting and scene segmentation, radiation correction, geometric correction, image orthographic projection, target detection and identification, ground feature classification and other processing plug-ins are integrated, and verification is carried out in a distributed cluster environment.
The invention designs a logic control service and master-slave node two-stage scheduling mechanism based on a workflow engine, realizes the control of the whole processing flow by a multi-stage gradual progressive scheduling mode, and realizes each stage of scheduling module based on the micro-service idea, thereby improving the scheduling flexibility; and a plug-in load balancing scheduling strategy is made by fully combining the characteristics of the remote sensing data processing plug-in.
In a word, the invention fully utilizes the flexible scheduling capability of the workflow technology, simplifies the development process, provides the WYSIWYG flow arrangement capability and realizes the distributed parallel scheduling capability of the complex remote sensing data processing.
Claims (4)
1. A distributed scheduling method of remote sensing data processing plug-ins based on a workflow engine is characterized by comprising the following steps:
(1) carrying out logic scheduling management on the execution process through a logic control service, arranging the whole operation flow, configuring the operation parameters and initiating the whole flow;
performing service plug-in management and service scheduling through the master node service, receiving parameter information to be executed transmitted by the logic control service, collecting execution result information and feeding back the execution result information to the logic control service, coordinating all computing resources in a cluster, executing a load balancing strategy and ensuring reasonable distribution of tasks;
receiving an execution task parameter sent by the master node service through the slave node service, calling and executing a relevant plug-in, and reporting an execution result and the use condition of slave node resources to the master node service;
(2) for the remote sensing data processing plug-in, the relevant information of the plug-in is described according to the unified template, the host node service catalogs and stores the plug-in, manages the plug-in a visual mode and provides functions of adding, deleting, modifying and searching the plug-in;
(3) the logic control service reads and processes the plug-in database table based on the workflow engine, analyzes the information of the stored processing plug-ins, organizes and manages various processing plug-ins in a visual mode, and finishes the flow arrangement and parameter transmission setting of the processing plug-ins in a dragging mode;
(4) after the process arrangement is finished, executing the process scheduling according to the process input parameters, wherein in the executing process, the logic control service determines task plug-ins needing to be executed next step according to the process arrangement result and transmits the parameters to the master node service, and the master node service generates calling parameters and sends the calling parameters to the slave node service needing to be scheduled according to the deployment condition of the processing plug-ins and the service condition of each slave node resource based on a polling scheduling strategy;
(5) and the slave node service generates a specific plug-in calling parameter according to the parameter information sent by the master node service, and completes the calling of the plug-in.
2. The distributed scheduling method of remote sensing data processing plug-in based on workflow engine as claimed in claim 1, wherein in step (1), the logic control service implements multiple execution strategies of merging, splitting, paralleling, asynchronizing and parameter pairing through the expansion and modification of workflow engine, and supports any combination of these execution strategies; and aiming at the same plug-in processing step in the processing process, service instantiation segmentation is carried out, so that the same plug-in can generate a plurality of service instances, and the instances are independent and can be processed in parallel.
3. The distributed scheduling method of remote sensing data processing plug-in based on workflow engine as claimed in claim 1, characterized in that in step (2), the resource information occupied by various plug-ins when executed is determined according to the service plug-in type, including the CPU core number and the memory capacity, and the relevant information of the plug-ins includes the name of the plug-ins, the function description of the plug-ins, the number of occupied CPU, the number of occupied memory, and the calling service interface of the plug-ins.
4. The distributed scheduling method of remote sensing data processing plug-ins based on workflow engine as claimed in claim 1, wherein in step (4), it is first clear in which slave nodes the plug-ins to be invoked are deployed, the master node service calculates according to the current resource usage reported by the slave node services, i.e. the number of idle CPUs of the slave nodes and the memory capacity, in combination with the resource information occupied by the processing plug-ins to be invoked when executed, and finally determines which node to invoke, the calculation formula is as follows:
C= Cn -Ca
M= Mn-Ma
cn is the total core number of the current polling slave node CPU, Ca is the core number occupied when the processing plug-in to be called is executed, Mn is the total memory amount of the slave node, and Ma is the memory capacity occupied when the processing plug-in to be called is executed; and when both M and C are larger than 0, determining to call the processing plug-in of the current slave node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210256586.2A CN114691233A (en) | 2022-03-16 | 2022-03-16 | Remote sensing data processing plug-in distributed scheduling method based on workflow engine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210256586.2A CN114691233A (en) | 2022-03-16 | 2022-03-16 | Remote sensing data processing plug-in distributed scheduling method based on workflow engine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114691233A true CN114691233A (en) | 2022-07-01 |
Family
ID=82140037
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210256586.2A Pending CN114691233A (en) | 2022-03-16 | 2022-03-16 | Remote sensing data processing plug-in distributed scheduling method based on workflow engine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114691233A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116541018A (en) * | 2023-06-19 | 2023-08-04 | 之江实验室 | Distributed model compiling system, method, device, medium and equipment |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107423053A (en) * | 2017-06-15 | 2017-12-01 | 东莞理工学院 | The webization model encapsulation and distributed approach of a kind of remote sensing image processing |
| CN108829509A (en) * | 2018-05-03 | 2018-11-16 | 山东汇贸电子口岸有限公司 | Distributed container cluster framework resources management method based on domestic CPU and operating system |
| EP3544260A1 (en) * | 2016-12-15 | 2019-09-25 | Huawei Technologies Co., Ltd. | Service layout method and device, and server |
| CN111142867A (en) * | 2019-12-31 | 2020-05-12 | 谷云科技(广州)有限责任公司 | Service visual arrangement system and method under micro-service architecture |
| CN111694888A (en) * | 2020-06-12 | 2020-09-22 | 谷云科技(广州)有限责任公司 | Distributed ETL data exchange system and method based on micro-service architecture |
| CN111858001A (en) * | 2020-07-15 | 2020-10-30 | 武汉众邦银行股份有限公司 | Workflow processing method based on micro-service architecture system |
| CN112445497A (en) * | 2020-11-25 | 2021-03-05 | 中国电子科技集团公司第五十四研究所 | Remote sensing image processing system based on plug-in extensible architecture |
| CN112506498A (en) * | 2020-11-30 | 2021-03-16 | 广东电网有限责任公司 | Intelligent visual API arrangement method, storage medium and electronic equipment |
| CN113010598A (en) * | 2021-04-28 | 2021-06-22 | 河南大学 | Dynamic self-adaptive distributed cooperative workflow system for remote sensing big data processing |
| CN113268319A (en) * | 2021-05-07 | 2021-08-17 | 中国电子科技集团公司第五十四研究所 | Business process customization and distributed process scheduling method based on micro-service architecture |
| CN113535362A (en) * | 2021-07-26 | 2021-10-22 | 北京计算机技术及应用研究所 | Distributed scheduling system architecture and micro-service workflow scheduling method |
| CN113626002A (en) * | 2021-08-13 | 2021-11-09 | 中国工商银行股份有限公司 | Service execution method and device |
-
2022
- 2022-03-16 CN CN202210256586.2A patent/CN114691233A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3544260A1 (en) * | 2016-12-15 | 2019-09-25 | Huawei Technologies Co., Ltd. | Service layout method and device, and server |
| CN107423053A (en) * | 2017-06-15 | 2017-12-01 | 东莞理工学院 | The webization model encapsulation and distributed approach of a kind of remote sensing image processing |
| CN108829509A (en) * | 2018-05-03 | 2018-11-16 | 山东汇贸电子口岸有限公司 | Distributed container cluster framework resources management method based on domestic CPU and operating system |
| CN111142867A (en) * | 2019-12-31 | 2020-05-12 | 谷云科技(广州)有限责任公司 | Service visual arrangement system and method under micro-service architecture |
| CN111694888A (en) * | 2020-06-12 | 2020-09-22 | 谷云科技(广州)有限责任公司 | Distributed ETL data exchange system and method based on micro-service architecture |
| CN111858001A (en) * | 2020-07-15 | 2020-10-30 | 武汉众邦银行股份有限公司 | Workflow processing method based on micro-service architecture system |
| CN112445497A (en) * | 2020-11-25 | 2021-03-05 | 中国电子科技集团公司第五十四研究所 | Remote sensing image processing system based on plug-in extensible architecture |
| CN112506498A (en) * | 2020-11-30 | 2021-03-16 | 广东电网有限责任公司 | Intelligent visual API arrangement method, storage medium and electronic equipment |
| CN113010598A (en) * | 2021-04-28 | 2021-06-22 | 河南大学 | Dynamic self-adaptive distributed cooperative workflow system for remote sensing big data processing |
| CN113268319A (en) * | 2021-05-07 | 2021-08-17 | 中国电子科技集团公司第五十四研究所 | Business process customization and distributed process scheduling method based on micro-service architecture |
| CN113535362A (en) * | 2021-07-26 | 2021-10-22 | 北京计算机技术及应用研究所 | Distributed scheduling system architecture and micro-service workflow scheduling method |
| CN113626002A (en) * | 2021-08-13 | 2021-11-09 | 中国工商银行股份有限公司 | Service execution method and device |
Non-Patent Citations (3)
| Title |
|---|
| 冯阳;张国强;: "基于工作流技术的遥感卫星数据接收调度系统的设计与实现", 无线电工程, no. 11, 19 October 2018 (2018-10-19) * |
| 承林;王海宁;高春成;: "分布式任务调度在电力市场交易系统中的应用设计", 计算机应用与软件, no. 11, 12 November 2018 (2018-11-12) * |
| 谭娟 等: "开放式遥感数据服务系统架构技术研究", 武汉大学学报· 信息科学版, 31 December 2015 (2015-12-31) * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116541018A (en) * | 2023-06-19 | 2023-08-04 | 之江实验室 | Distributed model compiling system, method, device, medium and equipment |
| CN116541018B (en) * | 2023-06-19 | 2023-09-15 | 之江实验室 | Distributed model compiling system, method, device, medium and equipment |
| US11934887B1 (en) | 2023-06-19 | 2024-03-19 | Zhejiang Lab | Distributed model compilation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101208695B (en) | Managing metadata for graph-based computations | |
| CN101887365B (en) | Method and system for constructing executable code for component-based applications | |
| CN1841379B (en) | Mapping of a file system model to a database object | |
| CN103412853B (en) | A kind of automatic example generation method for file convertor | |
| US7072898B2 (en) | Method and apparatus for exchanging communications between heterogeneous applications | |
| US20050160398A1 (en) | Method and apparatus for dataflow creation and execution | |
| CN101589366A (en) | Parallelization and instrumentation in a producer graph oriented programming framework | |
| US11194595B2 (en) | Generation apparatus, program, and generation method | |
| US20050108684A1 (en) | Method and system for generating an application object repository from application framework metadata | |
| CN101617292A (en) | Programming and execution towards producer graph | |
| CN101535955A (en) | Managing parameters for graph-based computations | |
| CN102043657A (en) | File serialization method of model library of physical modeling language Modelica | |
| CN104572895A (en) | MPP (Massively Parallel Processor) database and Hadoop cluster data intercommunication method, tool and realization method | |
| US11003635B2 (en) | Database scheme for storing generic data | |
| CN104424018A (en) | Distributed calculating transaction processing method and device | |
| CN110764752A (en) | System and method for realizing graphical service arrangement of Restful service based on micro-service architecture | |
| CN100511140C (en) | Method for script language calling multiple output parameter interface by component software system | |
| CN101408909B (en) | Method for describing product information model | |
| CN114691233A (en) | Remote sensing data processing plug-in distributed scheduling method based on workflow engine | |
| CN114594927A (en) | Low code development method, device, system, server and storage medium | |
| CN105468793A (en) | Automated management method for simulation model data | |
| CN102929853A (en) | DCS (Data Communication System) project data generating system and method based on Excel sheet association | |
| CN101794223B (en) | Design method of WADE service message architecture | |
| CN101894317A (en) | System and method for driving business logic through data changes | |
| CN110245688B (en) | Data processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |