CN115809145A - Control scheduling method and device for multi-source image processing flow - Google Patents
Control scheduling method and device for multi-source image processing flow Download PDFInfo
- Publication number
- CN115809145A CN115809145A CN202211602486.7A CN202211602486A CN115809145A CN 115809145 A CN115809145 A CN 115809145A CN 202211602486 A CN202211602486 A CN 202211602486A CN 115809145 A CN115809145 A CN 115809145A
- Authority
- CN
- China
- Prior art keywords
- task
- processing
- algorithm
- image data
- completed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A control scheduling method and device for multi-source image processing flow belongs to the image processing field, and is characterized by comprising the following steps: combing the naming rules of the acquired image data, and appointing the conditions for screening the image data; carrying out secondary distinguishing on the screened images through the identification to obtain image data to be processed; modularizing a processing algorithm for image data processing to generate a processing template; filling the image to be processed into a processing template to generate a task flow; after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding a task to be completed depending on the current task, inquiring whether other dependent tasks of the task to be completed meet a submission condition, if so, submitting, and otherwise, ending, and waiting for triggering submission by other dependent tasks; and sequentially circulating until all algorithm modules are completed. Automatic processing is the main, and manual intervention is for assisting, reduces artifical the participation under the unchangeable prerequisite of work load, can guarantee the data processing precision simultaneously, the cost of using manpower sparingly.
Description
Technical Field
The invention belongs to the field of image processing, and particularly relates to a control scheduling method and device for a multi-source image processing flow.
Background
The rapid development of science and technology brings performance improvement of hardware equipment and optimization of software functions and algorithms, and has the same great influence on the field of earth observation. Various telemetry devices such as satellites, aviation airplanes, drones, etc. can produce images of various types, number, and sizes, which all present significant opportunities and challenges for image processing. Due to the characteristics of remote sensing image data, the performance requirement of the processing process on the computer is high, and the processing efficiency can be obviously improved by using the existing high-performance computing framework. However, the existing HPC (high performance computing) scheduling framework, such as HTCondor, supports task flow, and uses genetic relationships to describe the order of algorithm execution, and all task execution is based on file description; SLURM does not support task flow scheduling, and if the task flow needs to be submitted, the dependency relationships are sequentially established when the task flow needs to be submitted. After the tasks are submitted, the two types of tasks can automatically complete algorithm task execution, but the task arrangement difficulty is increased and the manual participation degree is too high due to the combination of industrial characteristics and large batch of data; when a task with a complex flow is met, manual intervention cannot be performed in the operation process, and potential problems are corrected in time. Therefore, considering the business background, it is necessary to design a new scheduling and control method. The existing image processing system has the following defects:
1. the increase of data volume leads to difficult task arrangement; each link of image processing needs manual participation, and the labor cost investment is large; meanwhile, a large amount of repeated work brings bad work experience to operators, and subjective activity of people cannot be played in a key way;
2. the processing process has high coupling, and the processing flow is changeable when the image data processing is performed for different loads; the user is troublesome in formulating the flow according to the self requirement, and the flexibility is insufficient; the integration of new algorithms is complicated and difficult to debug. Various problems cause that the processing task is difficult to design and complete in time, so that the final product delivery faces risks;
3. the unreasonable use of computer resources, such as the timing performance of algorithm execution, the high efficiency of resource matching scheduling, and the controllability of algorithm flow execution, all cause waste of hardware resources.
The overall treatment efficiency is greatly restricted by the lag of each link in the treatment process like the barrel effect.
Disclosure of Invention
The present invention is directed to solve the above problems, and provides a method and an apparatus for controlling and scheduling a multi-source image processing flow, which are capable of efficiently arranging image data and flexibly organizing the flow.
In a first aspect, the invention provides a control scheduling method for a multi-source image processing flow, which comprises the steps of combing the naming rule of the obtained image data, and appointing the condition for screening the image data; the condition for screening the image data comprises an identifier; the identifier is used for distinguishing the type of the image data and is present in a file name, such as MSS, PAN and the like; screening the images according to the screening conditions; secondly, carrying out secondary distinguishing on the screened images through the identification to finish the organization of image pairs and obtain image data to be processed; the identification is the identification of the image name and is used for distinguishing which type of image the identification belongs to;
modularizing processing algorithms for image data processing, and organizing the modularized processing algorithms in a directed acyclic graph manner; meanwhile, according to the dependency relationship of the processing algorithm, the association of the algorithm is completed, and a processing template is generated; filling the image to be processed into a processing template to generate a task flow;
forming a task file which supports the Sbatch submission of the SLURM according to the task parameter information and the configuration information included in the task flow; then generating a pre-execution (pre) task file and a post-execution (post) task file for each task file; for example, there is an algorithm a, and the submission flow of the algorithm becomes a _ pre- > a _ post, so that the relationship that a depends on a _ pre and a _ post depends on a is formed, and the three form a submission unit;
after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding all to-be-completed tasks depending on the current task, inquiring whether other dependent tasks of the to-be-completed tasks meet a submission condition, if so, submitting, and if not, finishing, and waiting for triggering submission by other dependent tasks; and sequentially circulating until all algorithm modules are completed.
Further, according to the control scheduling method of the multi-source image processing flow, when the processing algorithm is modularized, rules of input, output and variable parameters of the algorithm need to be formulated, and a task description file is formed.
Further, in the control scheduling method of the multi-source image processing flow, the conditions for screening data further include load and regular expression;
the load is the name of a sensor for acquiring image data and is used as an original input mark of a processing algorithm, such as panchromatic, multispectral and the like;
when the regular expression is written, the same parts of the same pair of images need to be matched, and different parts can pass through wildcards
Matching any character is the key to be able to screen out the desired image.
Furthermore, the control scheduling method of the multi-source image processing flow comprises a processing algorithm at least comprising an input path, an output path and resource parameters;
the resource parameters comprise the number of processes, the number of CPUs (central processing units), the number of GPUs (graphic processing units) and the size of a memory; the resource parameters are the basic basis for completing algorithm scheduling.
Further, according to the control scheduling method of the multi-source image processing flow, the processing algorithm is divided into one-to-one or one-to-many or many-to-one types according to the corresponding relation between the input data and the output data, so that data filling is performed when an algorithm task is organized, for example, a group of related inputs generate a group of related outputs, which is called as 'one-to-one', and the like.
Furthermore, the control scheduling method of the multi-source image processing flow of the invention organizes the image to be processed into an engineering file before the image to be processed is filled into the processing template to generate the task flow, and the engineering file is used as the original input of the task flow.
In a second aspect, the invention provides a control scheduling system for a multi-source image processing flow, which comprises a target image acquisition module, a processing algorithm modularization module and a task processing module;
the target image acquisition module is used for appointing conditions for screening the image data according to the naming rule of the acquired image data; the condition for screening the image data comprises an identification; the identification is used for distinguishing the type of the image data; screening the images according to the screening conditions; secondly, carrying out secondary distinguishing on the screened images through the identification to finish the organization of image pairs and obtain image data to be processed; the identification is the identification of the image name and is used for distinguishing which type of image the identification belongs to;
the processing algorithm modularization module is used for modularizing the processing algorithm and organizing the modularized processing algorithm in a directed acyclic graph mode; meanwhile, according to the dependency relationship of the processing algorithm, the association of the algorithm is completed, and a processing template is generated; filling the image to be processed into a processing template to generate a task flow;
the task processing module is used for forming a task file which supports the SLURM and is submitted by the sbatch according to the task parameter information and the configuration information which are included in the task flow; then generating a pre-execution task file and a post-execution task file for each task file;
the task processing module executes scheduling management, specifically: after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding all tasks to be completed which depend on the current task, then inquiring whether other dependent tasks of the tasks to be completed meet a submission condition, if so, submitting, and otherwise, finishing, and waiting for other dependent tasks to trigger submission; and sequentially circulating until all algorithm modules are completed.
Further, in the control scheduling system of the multi-source image processing flow, the conditions for screening data further include loads and regular expressions;
the load is the name of a sensor for acquiring image data and is used as an original input mark of a processing algorithm;
when the regular expression is written, the same parts of the same pair of images need to be matched, and different parts can pass through wildcards
To match any character.
In a third aspect, the present invention provides an electronic device for controlling and scheduling multi-source image processing flow, including a processor and a memory electrically connected to each other; the memory is used for storing a computer program; when the processor executes the computer program, the method for controlling and scheduling the multi-source image processing flow according to the first aspect may be implemented.
In a fourth aspect, the present invention provides a computer readable storage medium having a computer program stored thereon; when the computer program is executed, the method for controlling and scheduling the multi-source image processing flow according to the first aspect may be implemented.
The control scheduling method and the control scheduling device for the multi-source image processing flow have the following technical effects: (1) The automatic processing is mainly performed, the manual intervention is assisted, the manual participation is reduced on the premise of no change of workload, the data processing precision can be ensured, and the labor cost is saved;
(2) Decoupling algorithm development and platform development and algorithm development by independently defining a plug-in protocol; by processing the definition template, the operation threshold can be reduced, and the flexible and changeable task flow can be completed;
(3) Task flow scheduling is automatically completed by appointing computing resources, and stable and efficient operation of tasks can be guaranteed; meanwhile, abundant manual intervention interfaces are provided, so that the waste of computing resources can be avoided, and error correction can be carried out in time.
Drawings
FIG. 1 is a schematic diagram illustrating a process of configuring data to be processed according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a directed acyclic graph according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of a task generation process according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a process flow scheduling process according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a control scheduling system of a multi-source image processing flow according to an embodiment of the present invention.
Detailed Description
The following describes the control scheduling method and apparatus for multi-source image processing flow according to the present invention in detail with reference to the accompanying drawings and embodiments.
Example one
The embodiment discloses a control scheduling method of a multi-source image processing flow, which comprises the steps of combing the naming rule of the obtained image data, and appointing the condition for screening the image data; the condition for screening the image data comprises an identifier; the identification is used for distinguishing the type of the image data; screening the images according to the screening conditions; and secondly, carrying out secondary distinguishing on the screened images through the identification to finish the organization of the image pair and obtain image data to be processed.
When the processing algorithm is modularized, rules of input, output and variable parameters of the algorithm need to be formulated to form a task description file. The condition of screening data also comprises load and regular expression;
the load is the name of a sensor for acquiring image data and is used as an original input mark of a processing algorithm; when the regular expression is written, the same parts of the same pair of images need to be matched, and different parts can pass through wildcards
To match any character.
In the method for controlling and scheduling a multi-source image processing flow according to the embodiment of the present disclosure, as shown in fig. 1, a process of configuring data to be processed may be summarized as the following specific steps:
s11, selecting a top-level catalog of the data to be screened, wherein the catalog contains all the data to be screened;
s12, selecting configuration information of the target data, namely configuration rules containing loads, identifications and regular expressions, and completing configuration verification, deduplication and pairing operations through integration;
s13, traversing each file in the folder through breadth-first search;
s14, matching each file with the regular expression, if the matching is successful, storing the matching identifier in a standby state;
s15, if the matched identifier does not exist, waiting for pairing, and if the matched identifier exists, completing pairing with the existing data;
s16, finishing the engineering json organization after all the pairs are finished;
s17, taking the project file as input, and calculating the geographic range of the image data in the project;
and S18, performing secondary screening on the visual software according to the geographical position relationship to further clarify task contents.
Then modularizing processing algorithms for image data processing, and organizing the modularized processing algorithms in a directed acyclic graph manner; meanwhile, according to the dependency relationship of the processing algorithm, the association of the algorithm is completed, and a processing template is generated; and filling the image to be processed into a processing template to generate a task flow. The processing algorithm at least comprises an input path, an output path and resource parameters; the resource parameters comprise the number of processes, the number of CPUs (central processing units), the number of GPUs (graphic processing units) and the size of a memory; the processing algorithm is divided into one-to-one or one-to-many or many-to-one types according to the corresponding relation between the input data and the output data.
The time sequence of the algorithm can be well described by a directed acyclic graph shown in fig. 2. START and END identify the beginning and END of the flow, and after the flow is started, the algorithm A is executed first; B. the C algorithm depends on the A algorithm, and after the execution of the A algorithm is finished, the B, C algorithm starts to execute in parallel; the D algorithm depends on B, C algorithm, after the B, C algorithm is executed, the D algorithm is executed, and the whole process is finished when the D algorithm is executed. A user can easily complete the customized processing task only by completing the selection of the front and back dependent data of input and output as a graph dragging algorithm module, and the whole design is completely easy to integrate, maintain and expand. And the interior of the scheduling is stored by a dictionary data structure, so that all algorithm module tasks are organized in sequence through the corrected breadth traversal algorithm, and finally, the landing of the tasks to be completed is generated, and the basis for algorithm execution scheduling is provided.
In the method for controlling and scheduling a multi-source image processing flow according to the embodiment of the present disclosure, as shown in fig. 3, a process of generating a task flow may be summarized as the following specific steps:
s21, defining plug-in types, namely input and output corresponding relations, and writing algorithm plug-ins according to a plug-in protocol;
s22, uploading the plug-in unit to complete the verification of the plug-in unit parameters;
s23, determining a dependency relationship and an input-output relationship according to requirements;
step S24, determining input for each algorithm, such as raw data or the result of an intermediate algorithm;
s25, organizing the algorithm module into a DAG (directed acyclic graph) form, strictly forbidding the existence of a ring, otherwise, the process cannot exit;
s26, according to the sequential dependency relationship executed by the plug-ins, sequentially completing data filling of tasks through data organization and breadth traversal;
s27, generating one task or a plurality of task description files by an algorithm module according to the plug-in type; and meanwhile, according to the demand configuration of the plug-in, the generation of the scheduling configuration file is completed.
Finally, according to the task parameter information and the configuration information included in the task flow, a task file which supports the SLURM and is submitted by the sbatch is formed; then generating a pre-execution task file and a post-execution task file for each task file; during process scheduling, after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding all to-be-completed tasks depending on the current task, inquiring whether other dependent tasks of the to-be-completed tasks meet a submission condition, if so, submitting, and otherwise, ending, and waiting for other dependent tasks to trigger submission; and sequentially circulating until all algorithm modules are completed.
In particular implementations, it is also feasible to simply accomplish task scheduling directly per node in a DAG (directed acyclic graph) graph through dependencies, but this is well complemented by the lack of control over task flow execution, creating pre-execution (pre) and post-execution (post) tasks. For example, a breakpoint function is added in the pre-execution, the output of the previous task is checked through manual checking or interactive tools, and then whether the operation is continued, the parameter is updated to operate again, the parameter of the next task is modified, and the like, so that resources are saved, and the result quality is improved. The analysis of the module execution condition is added in the task of post-execution, the synchronization of the task information is completed according to the execution condition, whether the next module is submitted or not is determined, or the whole process is stopped, and the function of starting and stopping is achieved.
Meanwhile, the functions of pre-execution and post-execution are not limited to the above, and the functions of timing task, main task rollback, subtask rollback, pause, continuation, filing and the like can be integrated according to specific requirements in specific applications, so that the whole process of the whole process can be controlled. And a functional interface is provided to the outside through the REST API, and a user can also control details of each task execution through a visual interface. Meanwhile, the task scheduling granularity is determined by evaluating the task flow and splitting the task function, so that the tasks are enabled to be parallel as much as possible, the task scheduling efficiency is greatly improved, and the complex data processing process is clearer.
In the method for controlling and scheduling a multi-source image processing flow according to the embodiment of the present disclosure, as shown in fig. 4, a flow scheduling process may be summarized as the following specific steps:
step S31, after the task file is completed, automatically calling back a submission starting module;
step S32, the starting module completes task record creation and initializes the state of the algorithm module;
s33, inquiring all the modules to be submitted, traversing the states of the algorithm modules depended by all the algorithm modules to be submitted, and completing the submission of the pre-execution modules of the modules to be submitted only when the dependence is in a final state and is not failed;
step S34, performing checking task controllable information in advance, such as breakpoints, time delay and the like, after the checking is passed, registering a callback task, and submitting the callback task by a callback task completion algorithm module;
step S35, submitting the algorithm module, registering a callback task, and calling back a post-processing task of the current algorithm module;
step S36, after the post-processing is completed, returning to step S33;
s37, inquiring whether the task to be submitted is empty and whether the current task is empty, namely finishing task flow scheduling;
step S38, the task may execute the corresponding control function, such as pause, continue, rollback, continue, terminate, etc., in a state of running, failing, warning, etc., until the target precision product data is produced.
Example two
The control scheduling system of the multi-source image processing flow in the embodiment of the disclosure, as shown in fig. 5, includes a target image acquisition module, a processing algorithm modularization module and a task processing module;
the target image acquisition module is used for appointing conditions for screening the image data according to the naming rule of the acquired image data; the condition for screening the image data comprises an identifier; the identification is used for distinguishing the type of the image data; screening the images according to the screening conditions; secondly, carrying out secondary distinguishing on the screened images through the identification to finish the organization of image pairs and obtain image data to be processed; the identification is the identification of the image name and is used for distinguishing which type of image the identification belongs to; the condition of screening data also comprises load and regular expression; the load is the name of a sensor for acquiring image data and is used as an original input mark of a processing algorithm; when the regular expression is written, the same parts of the same pair of images need to be matched, and different parts can pass through wildcards
To match any character.
The processing algorithm modularization module is used for modularizing the processing algorithm and organizing the modularized processing algorithm in a directed acyclic graph mode; meanwhile, according to the dependency relationship of the processing algorithm, the association of the algorithm is completed, and a processing template is generated; filling the image to be processed into a processing template to generate a task flow;
the task processing module is used for forming a task file which supports the SLURM and is submitted by the sbatch according to the task parameter information and the configuration information which are included in the task flow; then generating a pre-execution task file and a post-execution task file for each task file;
the task processing module executes scheduling management, specifically: after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding all to-be-completed tasks depending on the current task, inquiring whether other dependent tasks of the to-be-completed tasks meet a submission condition, if so, submitting, and if not, finishing, and waiting for triggering submission by other dependent tasks; and sequentially circulating until all algorithm modules are completed.
EXAMPLE III
The embodiment of the disclosure discloses a control scheduling electronic device for a multi-source image processing flow, which comprises a processor and a memory which are electrically connected; the memory is used for storing a computer program; when the processor executes the computer program, the method for controlling and scheduling a multi-source image processing flow according to the first embodiment may be implemented, and the specific steps of the method for controlling and scheduling are the same as those of the first embodiment, and are not described herein again.
Example four
The disclosed embodiments disclose a computer-readable storage medium having a computer program stored thereon; when the computer program is executed, the method for controlling and scheduling a multi-source image processing flow according to the first embodiment can be implemented, and the specific steps of the method for controlling and scheduling are the same as those of the first embodiment, which is not described herein again.
The computer described in the embodiments of the present application may be a general purpose computer, a special purpose computer, a network of computers, or other programmable devices. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium. The computer readable storage medium may be any available medium that can be read by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital Versatile Disk (DVD)), or a semiconductor medium (e.g., solid State Disk (SSD)), among others. The software formed by the computer stored code can be located in the storage media mature in the field of random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, etc.
The functional modules in the embodiments of the present application may be integrated into one processing unit or module, or each module may exist alone physically, or two or more modules are integrated into one unit or module. In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are realized in whole or in part.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A control scheduling method for multi-source image processing flow is characterized by comprising the following steps:
combing the naming rules of the acquired image data, and appointing the conditions for screening the image data; the condition for screening the image data comprises an identifier; the identification is used for distinguishing the type of the image data; screening the images according to the screening conditions; secondly, carrying out secondary distinguishing on the screened images through the identification to finish the organization of image pairs and obtain image data to be processed;
modularizing processing algorithms for image data processing, and organizing the modularized processing algorithms in a directed acyclic graph manner; meanwhile, according to the dependency relationship of the processing algorithm, the association of the algorithm is completed, and a processing template is generated; filling the image to be processed into a processing template to generate a task flow;
forming a task file which supports the Sbatch submission of the SLURM according to the task parameter information and the configuration information included in the task flow; then generating a pre-execution task file and a post-execution task file for each task file;
after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding all tasks to be completed which depend on the current task, then inquiring whether other dependent tasks of the tasks to be completed meet a submission condition, if so, submitting, and otherwise, finishing, and waiting for other dependent tasks to trigger submission; and sequentially circulating until all algorithm modules are completed.
2. The method of claim 1, wherein: when the processing algorithm is modularized, rules of input, output and variable parameters of the algorithm need to be formulated to form a task description file.
3. The method of claim 1, wherein: the condition of screening data also comprises load and regular expression;
the load is the name of a sensor for acquiring image data and is used as an original input mark of a processing algorithm;
when the regular expression is written, the same parts of the same pair of images need to be matched, and different parts can pass through wildcards
To match any character.
4. The method of claim 2, wherein: the processing algorithm at least comprises an input path, an output path and resource parameters;
the resource parameters comprise the number of processes, the number of CPUs, the number of GPUs and the size of a memory.
5. The method of claim 4, wherein: the processing algorithm is divided into one-to-one or one-to-many or many-to-one types according to the corresponding relation between the input data and the output data.
6. The method of claim 1, wherein: before the image to be processed is filled into the processing template to generate the task flow, the image to be processed is organized into an engineering file, and the form of the engineering file is used as the original input of the task flow.
7. A control scheduling system of multi-source image processing flow is characterized in that: the system comprises a target image acquisition module, a processing algorithm modularization module and a task processing module;
the target image acquisition module is used for appointing conditions for screening the image data according to the naming rule of the acquired image data; the condition for screening the image data comprises an identifier; the identification is used for distinguishing the type of the image data; screening the images according to the screening conditions; secondly, carrying out secondary distinguishing on the screened images through the identification to finish the organization of image pairs and obtain image data to be processed; the identification is the identification of the image name and is used for distinguishing which type of image the identification belongs to;
the processing algorithm modularization module is used for modularizing the processing algorithm and organizing the modularized processing algorithm in a directed acyclic graph mode; meanwhile, according to the dependency relationship of the processing algorithm, the association of the algorithm is completed, and a processing template is generated; filling the image to be processed into a processing template to generate a task flow;
the task processing module is used for forming a task file which supports the SLURM and is submitted by the sbatch according to the task parameter information and the configuration information which are included in the task flow; then generating a pre-execution task file and a post-execution task file for each task file;
the task processing module executes scheduling management, specifically: after one task is completed, traversing node tasks of the directed acyclic graph by using a traversal algorithm, finding all to-be-completed tasks depending on the current task, inquiring whether other dependent tasks of the to-be-completed tasks meet a submission condition, if so, submitting, and if not, finishing, and waiting for triggering submission by other dependent tasks; and sequentially circulating until all algorithm modules are completed.
8. The system of claim 7, wherein: the condition of screening data also comprises load and regular expression;
the load is the name of a sensor for acquiring image data and is used as an original input mark of a processing algorithm;
when the regular expression is written, the same parts of the same pair of images need to be matched, and different parts can pass through wildcards
To match any character.
9. A control scheduling electronic device of multi-source image processing flow comprises a processor and a memory which are electrically connected; the memory is used for storing a computer program; the method is characterized in that: when the processor executes the computer program, the method for controlling and scheduling the multi-source image processing flow according to any one of claims 1 to 6 can be realized.
10. A computer-readable storage medium characterized by: the storage medium having stored thereon a computer program; the computer program, when executed, may implement the method of controlling and scheduling multi-source image processing flow according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211602486.7A CN115809145A (en) | 2022-12-14 | 2022-12-14 | Control scheduling method and device for multi-source image processing flow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211602486.7A CN115809145A (en) | 2022-12-14 | 2022-12-14 | Control scheduling method and device for multi-source image processing flow |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115809145A true CN115809145A (en) | 2023-03-17 |
Family
ID=85485799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211602486.7A Pending CN115809145A (en) | 2022-12-14 | 2022-12-14 | Control scheduling method and device for multi-source image processing flow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115809145A (en) |
-
2022
- 2022-12-14 CN CN202211602486.7A patent/CN115809145A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3678068A1 (en) | Distributed system for executing machine learning and method therefor | |
| CN108280023B (en) | Task execution method and device and server | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| AU2017327824B2 (en) | Data integration job conversion | |
| CN111026568B (en) | Data and task relation construction method and device, computer equipment and storage medium | |
| US20140101213A1 (en) | Computer-readable recording medium, execution control method, and information processing apparatus | |
| CN111475137A (en) | Method, system and equipment for predicting software development requirements | |
| CN111459621A (en) | Cloud simulation integration and scheduling method and device, computer equipment and storage medium | |
| US11782888B2 (en) | Dynamic multi-platform model generation and deployment system | |
| JP2023553220A (en) | Process mining for multi-instance processes | |
| CN111143390A (en) | Method and device for updating metadata | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN113190576A (en) | Data processing method and device, computer equipment and readable storage medium | |
| US11995587B2 (en) | Method and device for managing project by using data merging | |
| CN115809145A (en) | Control scheduling method and device for multi-source image processing flow | |
| CN112367205B (en) | Processing method and scheduling system for HTTP scheduling request | |
| CN107526573B (en) | Method for processing remote sensing image by adopting parallel pipeline | |
| CN112966974A (en) | Project configuration method, device, equipment and medium | |
| EP4109364B1 (en) | Method and device for managing project by using data pointer | |
| CN108595552A (en) | Data cube dissemination method, device, electronic equipment and storage medium | |
| EP4109365A1 (en) | Method and device for managing project by using cost payment time point setting | |
| CN109814991A (en) | A kind of data administer in task management method and device | |
| EP4109367A1 (en) | Method and device for managing project by using data filtering | |
| CN117271011B (en) | Conversion processing system and method for building information model data | |
| WO2020259020A1 (en) | Instruction block processing method and apparatus, storage medium, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |