CN109857462B - Background Docker task mapping method of remote sensing image visual editor - Google Patents
Background Docker task mapping method of remote sensing image visual editor Download PDFInfo
- Publication number
- CN109857462B CN109857462B CN201910073457.8A CN201910073457A CN109857462B CN 109857462 B CN109857462 B CN 109857462B CN 201910073457 A CN201910073457 A CN 201910073457A CN 109857462 B CN109857462 B CN 109857462B
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- sensing image
- image processing
- docker
- jenkins
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Image Processing (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a background Docker task mapping method of a remote sensing image visual editor, which designs a remote sensing image processing visual editor at the front end, and a user only needs to interact with the remote sensing image processing visual editor through a browser to automatically edit a required remote sensing processing flow; the method comprises the steps that a JSON file transmitted by a front end is received by the rear end, the JSON file is analyzed, a Jenkins Pipeline DSL script part which can be scheduled and executed by Jenkins is automatically translated according to JSON description semantics, a config.
Description
Technical Field
The invention relates to the field of remote sensing image processing methods, in particular to a background Docker task mapping method of a remote sensing image visual editor.
Background
The remote sensing image processing comprises the processing steps of preprocessing of the remote sensing image, geometric correction, image enhancement, image cutting, image mosaic, color equalization and the like. These processing steps correspond to different processing algorithms for the remote sensing image. Due to the mass of the remote sensing images, when the remote sensing images are processed, the resource utilization rate is improved by fully utilizing parallel computation, the computation time is shortened, and the method has important significance for processing the remote sensing images. The Docker is an open-source container application engine, so that a developer can pack development environment, codes, configuration files and the like into the container and distribute the container to various platforms such as Windows, Linux, Mac and the like, and cross-platform scheduling of application programs can be realized based on the Docker technology. In addition, the Docker technique is different from the minute-level start-up speed of a virtual machine, and the Docker container start-up speed can be realized in the second level. Moreover, the Docker container has little demand on system resources, thousands of Docker containers can be run on one host at the same time, and generally one host only supports dozens of virtual machines. Meanwhile, the application programs running on different Dockers are also isolated from each other through the containers, so that the utilization rate of the physical resources of the host can be greatly improved. Jenkins is continuous integrated open source software developed based on Java, can be used for distributed management and trigger execution of tasks, and can monitor the running state of the tasks. Jenkins Pipeline DSL (domain-specific language) provides a set of grammar definitions which support the definition of a continuously integrated delivery Pipeline in Jenkins, and each processing step in the delivery Pipeline can be abstracted into a code module in the DSL grammar by a user through the Jenkins Pipeline DSL grammar, so that the user can carry out modeling in a mode of the code module. Jenkins Pipeline DSL syntax can integrate the Docker container, and through the analysis to Jenkins Pipeline DSL syntax description, realize that the task runs on different Dockers, but this set of Jenkins Pipeline DSL has a technical threshold of studying and mastering a new language to the remote sensing user, consequently need provide a visual interface for the user at the front end, let the user just can edit required remote sensing image processing flow through the mode of dragging, the remote sensing image processing flow that the user edited out simultaneously maps to different Dockers at the rear end and handles, has fine using value.
Disclosure of Invention
The invention provides a background Docker task mapping method for a remote sensing image visual editor, which can improve the utilization rate of resources.
In order to achieve the technical effects, the technical scheme of the invention is as follows:
a background Docker task mapping method of a remote sensing image visualization editor comprises the following steps:
s1: selecting keywords capable of meeting the requirement of describing a remote sensing image processing service process according to keywords given by a basic building block of Jenkins Pipeline hierarchical grammar according to the remote sensing image processing service description requirement;
s2: according to the keywords, defining a description method of the remote sensing image processing business process;
s3: designing an interface composition element and an attribute value of front-end visual interaction of a remote sensing image processing service flow;
s4: designing a pipeline component operation method for front-end visual interaction of a remote sensing image processing business process;
s5: installing a Docker engine in a slave node managed by Jenkins by adopting a machine of a Docker container operation task, and simultaneously recording an IP address provided with the Docker engine in a database table;
s6: when a user triggers and customizes a new remote sensing image processing flow, a foreground acquires a current timestamp, the timestamp is used as a current remote sensing image processing flow id number, a remote sensing image processing service flow editor is developed, and after the user customizes, a save button is used for sending a JSON file formed by visual interaction to a background;
s7: extracting nodes and attribute data in the JSON file according to the definition of a remote sensing image processing flow, constructing a DAG graph according to flow description, analyzing and traversing the DAG graph, automatically translating a Jenkins task logic control description flow in a Jenkins Pipeline DSL syntax format, reading a data table, and synchronizing information in the flow into the data table;
s8: and writing the Jenkins Pipeline DSL statement analyzed in the step S7 into a jobb config.xml file, and then creating a corresponding jobb by taking the generated config.xml file as a parameter in the Jenkins Pipeline api, so as to realize that the remote sensing processing steps are distributed into different Docker containers for processing according to parallel and serial processing steps at the back end.
Further, the specific process of step S1 is:
s11: analyzing the semantics represented by the keywords given by Jenkins Pipeline classification, and understanding the value range of the keywords;
s12: analyzing and describing the expressed service range required by the remote sensing image processing service flow;
s13: the minimum keyword set required by the extracted remote sensing image processing business process comprises the following steps: agent, locations, stage, steps, pipeline, parallel, build;
s14: and determining the value range of the keywords required for describing the remote sensing image processing service flow for front-end interaction.
Further, the specific process of step S2 is:
using agent to describe the execution position of the remote sensing processing step, using agent parameter as any to express that the execution can be executed on any machine, using docker to express that the remote sensing processing step is executed in docker, using always Pull to express that the mirror image is continuously downloaded from docker warehouse in docker parameter, using args to express the directory between the virtual machine and the docker, directly mounting the remote sensing image processing data into docker for processing, using customWorkspace to express the directory executed by the remote sensing image processing step, using image to express the name of the docker mirror image currently executing the remote sensing image processing task, and using label to express that the current docker container is operated on the node by Jenkins represented by label;
wherein, the keywords represent processing actions which should be contained in the whole remote sensing image processing flow, and the keywords can contain actions which are contained in a plurality of remote sensing image processing sub-flows stage; stage represents a description part for executing the remote sensing image processing step, and comprises steps parameters for specifically executing the remote sensing image processing step; steps is an execution command series for executing the remote sensing image processing step in the stage; pipeline represents an entrance mark of the whole remote sensing image processing flow; parallel indicates that the stage contained in it is a parallel execution relation; the job name is added after the build keyword to indicate the task name for starting the remote sensing image processing step.
Further, the specific process of step S3 is:
s31: designing a process name of a current remote sensing image processing process, generating a new id number of the whole remote sensing processing process according to a timestamp, and judging different remote sensing image processing processes;
s32: representing a remote sensing processing step by using nodes;
s33: representing the execution sequence between the nodes by using connecting lines between the nodes;
s34: one node has N predecessors and N successors, wherein N is more than or equal to 0;
s35: each node mainly contains the following attribute information: the method comprises the steps of processing a remote sensing image, wherein the current node represents a remote sensing image processing sub-process id, a current remote sensing image processing step name, a current remote sensing image processing step id value, a current remote sensing image processing step state, a subsequent remote sensing image processing step id value, a property used for recording a job and running information selected by a user, a machine used for recording an execution machine selected by the user, and a parallel used for recording whether the current node is a parallel node;
if the current node is not the sub-process, the current node is represented by-1, otherwise, a new remote sensing image processing sub-process id value is generated according to the timestamp; and a mark is used for indicating whether the current remote sensing processing step comprises a sub remote sensing image processing flow.
Further, the specific process of step S4 is:
s41: in order to enable a user to customize a remote sensing image processing flow, defining that pipeline can add nodes, delete nodes, edit nodes, add connecting lines between nodes, delete connecting lines between nodes, and analyze a remote sensing processing step to be processed;
s42: and a node is used for representing the sub pipeline, the sub pipeline can be edited by clicking the node, and the content in the sub pipeline can be hidden and expanded.
Further, the specific process of step S5 is:
s51: observing the health condition of a Docker engine machine to be deployed by using a node management page of Jenkins, and allocating a label to the current machine, wherein the label is used for identifying which machine is adopted to execute the Docker in a front-end process;
s52: remotely logging in the selected machine, and installing a Docker engine on the slave node to construct a Docker container environment capable of using remote sensing professional processing;
s53: the label value of the current machine is recorded in a database table.
Further, the specific process of step S6 is:
s61: calling a timestamp generation method to obtain a function method accurate to millisecond, and taking the function method as an id number of a remote sensing image processing flow;
s62: the front end sends a request to the back end to obtain a jobname and a jobid corresponding to the remote sensing image processing step established in the current Jenkins;
s63: developing a remote sensing service process editor, wherein the editor uses nodes to represent remote sensing image processing steps, can show a job name and a job id corresponding to the remote sensing image processing steps established in the current Jenkins for a user, and is used for selecting the remote sensing image processing steps integrated in the Jenkins for the user;
s64: the remote sensing service process editor can record the remote sensing image processing service process and the service of the attribute value of each node according to the visual customization result of the user, and can record the remote sensing image processing service process and the id number of the remote sensing image processing process so as to form a JSON description file of the remote sensing image processing process example;
s65: and when the user clicks the Save button, sending an http request and sending the JSON file of the front end to the back end.
Further, the specific process of step S7 is:
s71: the rear end receives a front end http request by using spring @ annotation and receives a JSON file sent by the front end;
s72: extracting nodes, relationships among the nodes and attribute values of the nodes according to a JSON-described remote sensing image processing flow, describing the nodes in the flow by using vertexes in a DAG (direct current) graph, describing the flow relationships among the nodes by using edges in the DAG graph, wherein the attribute values of the vertexes represent the attributes of the nodes;
s73: traversing a DAG graph, and gradually translating a Jenkins task logic control description flow in a Jenkins Pipeline DSL syntax format according to a description method of a remote sensing image processing service flow defined by two steps of keywords agent, sites, stage, steps, Pipeline, parallel and build;
s74: when the JSON file is analyzed and the keyword agent in the member object attribute value in the JSON is found to contain a docker parameter, the fact that the remote sensing image processing step corresponding to the current process needs to be mapped into the docker to be executed is indicated;
s75: when analyzing the Docker grammar mapping, if the user front end selects to download the mirror image from the warehouse, adding alwaysPull statement in Jenkins Pipeline DSL Docker statement, if the user adds the directory mounted between the virtual machine and the Docker at the front end, adding statements args '/mounted directory,/mounted directory ', if the user selects to define the working interval, adding the statement cumStomWorkspace ' user-defined working interval ', and when the user selects to pull the mirror image from the warehouse, adding a statement image ' RemoteSensingProcessing according to the mirror image name input by the user at the front end page, adding a statement label ' nodeX ' according to the label name input by the user at the front end page, and specifying the label of the running machine;
s76: according to the traversal translation result, respectively splicing DSL statements to obtain Jenkins Pipeline DSL statements;
s77: and reading the data table, judging whether the current data table contains the id value of the remote sensing image processing flow, if not, adding a new remote sensing image processing flow record, and if the data table is stored, refreshing the existing data record.
Further, the specific process of step S8 is:
s81: generating a new jobb by using the analyzed Jenkins Pipeline DSL according to a template for creating a jobb configuration.
S82: taking the generated jobfile in the xml as a parameter, and calling crete jobapi in Jenkins api to create a remote sensing image processing flow task in Jenkins;
s83: after a user triggers a remote sensing image processing flow execution button at the front end, a request is sent to the rear end, and the rear end calls a jenkins api to trigger a jenkins jobcorresponding to the current remote sensing image processing flow to execute a task;
s84: and (3) analyzing the jobconfig. xml file by Jenkins, and distributing the remote sensing processing steps into different Docker containers or Jenkins slave nodes for processing according to parallel and serial processing steps.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
1. the method is high in learning cost, so that a visual interface needs to be provided for a user at the front end, the user can edit the required remote sensing image processing flow in a dragging mode, the remote sensing image processing flow edited by the user is mapped to different Dockers at the rear end for processing, and the user can map the remote sensing image processing flow to the Dockers cluster for distributed parallel processing without knowing the morphology and grammar knowledge of Jenkins Pipeline DSL;
2. by adopting the method and the device, the remote sensing image processing flow can be rapidly and visually customized, the remote sensing image processing flow can be automatically mapped into the Docker cluster for distributed parallel processing, the utilization rate of resources can be greatly improved, and the remote sensing image processing flow can be visually customized by a user without developing other parallel and distributed scheduling modules.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the patent;
for the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;
it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
Example 1
As shown in fig. 1, a background Docker task mapping method for a remote sensing image visualization editor includes the following steps:
step one, according to the remote sensing image processing service description requirement, selecting keywords capable of meeting the description of the remote sensing image processing service process according to keywords given by a basic building block of Jenkins Pipeline hierarchical syntax;
the specific implementation manner of the first step is as follows:
1) analyzing the semantics represented by the keywords given by Jenkins Pipeline classification, and understanding the value range of the keywords;
2) analyzing and describing the expressed service range required by the remote sensing image processing service flow;
3) the minimum keyword set required by the extracted remote sensing image processing business process comprises the following steps: agent, locations, stage, steps, pipeline, parallel, build;
4) and determining the value range of the keywords required for describing the remote sensing image processing service flow for front-end interaction.
Step two: according to the keywords, defining a description method of the remote sensing image processing business process;
1) using agent to describe the execution position of the remote sensing processing step, using agent parameter as any to express that the remote sensing processing step can be executed on any machine, using docker to express that the remote sensing processing step is executed in docker, using always Pull to express that the mirror image is continuously downloaded from docker warehouse in docker parameter, using args to express the directory between the virtual machine and the docker, directly mounting the remote sensing image processing data into docker for processing, using customWorkspace to express the directory executed by the remote sensing image processing step, using image to express the name of the docker mirror image currently executing the remote sensing image processing task, and using label to express that the current docker container is operated on the node by Jenkins represented by label;
2) the keywords represent processing actions which should be contained in the whole remote sensing image processing flow, and the keywords can contain actions covered by a plurality of remote sensing image processing sub-flow processes;
3) stage represents a description part for executing the remote sensing image processing step, and comprises steps parameters for specifically executing the remote sensing image processing step;
4) steps is an execution command series for executing the remote sensing image processing step in the stage;
5) pipeline represents an entrance mark of the whole remote sensing image processing flow;
6) parallel indicates that the stage contained in it is a parallel execution relation;
7) the job name is added after the build keyword to indicate the task name for starting the remote sensing image processing step.
Designing interface composition elements and attribute values of front-end visual interaction of the remote sensing image processing business process;
1) designing a process name of a current remote sensing image processing process, generating a new id number of the whole remote sensing processing process according to a timestamp, and judging different remote sensing image processing processes;
2) representing a remote sensing processing step by using nodes;
3) representing the execution sequence between the nodes by using connecting lines between the nodes;
4) a node may have N (N is greater than or equal to 0) predecessors and N (N is greater than or equal to 0) successors;
5) each node mainly contains the following attribute information: the method comprises the steps of obtaining a remote sensing image processing sub-process id represented by a current node (if the current node is not a sub-process, the current node is represented by-1, otherwise, a new remote sensing image processing sub-process id value is generated according to a timestamp), a current remote sensing image processing step name, a current remote sensing image processing step id value, a current remote sensing image processing step state (whether the current remote sensing processing step contains a sub remote sensing image processing process is represented by a mark), a subsequent remote sensing image processing step id value, property used for recording a job and operation information selected by a user, a machine used for recording an execution machine selected by the user, and a parallel node;
designing a pipeline operation method of front-end visual interaction of the remote sensing image processing business process;
1) in order to enable a user to customize a remote sensing image processing flow, nodes can be added, nodes can be deleted, nodes can be edited, connecting lines can be added between the nodes, connecting lines can be deleted between the nodes, the remote sensing processing step 2) to be processed is analyzed, a node is used for representing a sub-pipeline, the sub-pipeline can be edited by clicking the node, and the content in the sub-pipeline can be hidden and expanded.
Step five: installing a Docker engine in a slave node managed by Jenkins by adopting a machine of a Docker container operation task, and recording the machine provided with the Docker engine in a database table;
1) observing the health condition of a Docker engine machine to be deployed by using a node management page of Jenkins, and allocating a label to the current machine, wherein the label is used for identifying which machine is adopted to execute the Docker in a front-end process;
2) remotely logging in the selected machine, and installing a Docker engine on the slave node to construct a Docker container environment capable of using remote sensing professional processing; 3) recording the label value of the current machine in a database table;
when a user triggers and customizes a new remote sensing image processing flow, a foreground acquires a current timestamp, the timestamp is used as a current remote sensing image processing flow id number, a remote sensing image processing service flow editor is developed, and after the user customizes, a save button is used for sending a JSON file formed by visual interaction to a background;
1) calling a timestamp generation method to obtain a function method accurate to millisecond, and taking the function method as an id number of a remote sensing image processing flow;
2) the front end sends a request to the back end to obtain a jobname and a jobid corresponding to the remote sensing image processing step established in the current Jenkins;
3) developing a remote sensing service process editor, wherein the editor uses nodes to represent remote sensing image processing steps, can show a job name and a job id corresponding to the remote sensing image processing steps established in the current Jenkins for a user, and is used for selecting the remote sensing image processing steps integrated in the Jenkins for the user;
4) the remote sensing service process editor can record the remote sensing image processing service process and the service of the attribute value of each node according to the visual customization result of the user, and can record the remote sensing image processing service process and the id number of the remote sensing image processing process so as to form a JSON description file of the remote sensing image processing process example;
5) and when the user clicks the Save button, sending an http request and sending the JSON file of the front end to the back end.
Extracting nodes and attribute data in the JSON file according to the definition of the remote sensing image processing flow, constructing a DAG graph according to the flow description, analyzing and traversing the DAG graph, automatically translating a Jenkins task logic control description flow in a Jenkins Pipeline DSL syntax format, reading a data table, and synchronizing information in the flow into the data table;
1) the rear end receives a front end http request by using spring @ annotation and receives a JSON file sent by the front end;
2) extracting nodes, relationships among the nodes and attribute values of the nodes according to a JSON-described remote sensing image processing flow, describing the nodes in the flow by using vertexes in a DAG (direct current) graph, describing the flow relationships among the nodes by using edges in the DAG graph, wherein the attribute values of the vertexes represent the attributes of the nodes;
3) traversing a DAG graph, and gradually translating a Jenkins task logic control description flow in a Jenkins Pipeline DSL syntax format according to a description method of a remote sensing image processing service flow defined by two steps of keywords agent, sites, stage, steps, Pipeline, parallel and build;
4) when the JSON file is analyzed and the keyword agent in the member object attribute value in the JSON is found to contain a docker parameter, the fact that the remote sensing image processing step corresponding to the current process needs to be mapped into the docker to be executed is indicated;
5) when analyzing the Docker grammar mapping, if the user front end selects to download the mirror image from the warehouse, adding alwaysPull statement in Jenkins Pipeline DSL Docker statement, if the user adds the directory mounted between the virtual machine and the Docker at the front end, adding statements args '/mounted directory,/mounted directory ', if the user selects to define the working interval, adding the statement cumStomWorkspace ' user-defined working interval ', and when the user selects to pull the mirror image from the warehouse, adding a statement image ' RemoteSensingProcessing according to the mirror image name input by the user at the front end page, adding a statement label ' nodeX ' according to the label name input by the user at the front end page, and specifying the label of the running machine;
6) according to the traversal translation result, respectively splicing DSL statements to obtain Jenkins Pipeline DSL statements;
7) and reading the data table, judging whether the current data table contains the id value of the remote sensing image processing flow, if not, adding a new remote sensing image processing flow record, and if the data table is stored, refreshing the existing data record.
Step eight: and writing the Jenkins Pipeline DSL statement analyzed in the step eight into a jobb config.xml file, and then creating a corresponding jobb by taking the produced config.xml file as a parameter in the Jenkins create jobapi, so that the remote sensing processing steps are distributed into different Docker containers for processing according to parallel and serial processing steps at the back end.
1) Generating a new jobb by using the analyzed Jenkins Pipeline DSL according to a template for creating a jobb configuration.
2) Taking the generated jobfile in the xml as a parameter, and calling crete jobapi in Jenkins api to create a remote sensing image processing flow task in Jenkins;
3) after a user triggers a remote sensing image processing flow execution button at the front end, a request is sent to the rear end, and the rear end calls a jenkins api to trigger a jenkins jobcorresponding to the current remote sensing image processing flow to execute a task;
4) and (3) analyzing the jobconfig. xml file by Jenkins, and distributing the remote sensing processing steps into different Docker containers or Jenkins slave nodes for processing according to parallel and serial processing steps.
The same or similar reference numerals correspond to the same or similar parts;
the positional relationships depicted in the drawings are for illustrative purposes only and are not to be construed as limiting the present patent;
it should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (9)
1. A background Docker task mapping method of a remote sensing image visualization editor is characterized by comprising the following steps:
s1: selecting keywords capable of meeting the requirement of describing a remote sensing image processing service process according to keywords given by a basic building block of Jenkins Pipeline hierarchical grammar according to the remote sensing image processing service description requirement;
s2: according to the keywords, defining a description method of the remote sensing image processing business process;
s3: designing an interface composition element and an attribute value of front-end visual interaction of a remote sensing image processing service flow;
s4: designing a pipeline component operation method for front-end visual interaction of a remote sensing image processing business process;
s5: installing a Docker engine in a slave node managed by Jenkins by adopting a machine of a Docker container operation task, and simultaneously recording an IP address provided with the Docker engine in a database table;
s6: when a user triggers and customizes a new remote sensing image processing flow, a foreground acquires a current timestamp, the timestamp is used as a current remote sensing image processing flow id number, a remote sensing image processing service flow editor is developed, and after the user customizes, a save button is used for sending a JSON file formed by visual interaction to a background;
s7: extracting nodes and attribute data in the JSON file according to the definition of a remote sensing image processing flow, constructing a DAG graph according to flow description, analyzing and traversing the DAG graph, automatically translating a Jenkins task logic control description flow in a Jenkins Pipeline DSL syntax format, reading a data table, and synchronizing information in the flow into the data table;
s8: and writing the Jenkins Pipeline DSL statement analyzed in the step S7 into a jobb config.xml file, and then creating a corresponding jobb by taking the generated config.xml file as a parameter in the Jenkins Pipeline api, so as to realize that the remote sensing processing steps are distributed into different Docker containers for processing according to parallel and serial processing steps at the back end.
2. The background Docker task mapping method for the remote sensing image visualization editor according to claim 1, wherein the specific process of step S1 is:
s11: analyzing the semantics represented by the keywords given by Jenkins Pipeline classification, and understanding the value range of the keywords;
s12: analyzing and describing the expressed service range required by the remote sensing image processing service flow;
s13: the minimum keyword set required by the extracted remote sensing image processing business process comprises the following steps: agent, locations, stage, steps, pipeline, parallel, build;
s14: and determining the value range of the keywords required for describing the remote sensing image processing service flow for front-end interaction.
3. The background Docker task mapping method for the remote sensing image visualization editor according to claim 2, wherein the specific process of step S2 is:
using agent to describe the execution position of the remote sensing processing step, using agent parameter as any to express that the execution can be executed on any machine, using docker to express that the remote sensing processing step is executed in docker, using always Pull to express that the mirror image is continuously downloaded from docker warehouse in docker parameter, using args to express the directory between the virtual machine and the docker, directly mounting the remote sensing image processing data into docker for processing, using customWorkspace to express the directory executed by the remote sensing image processing step, using image to express the name of the docker mirror image currently executing the remote sensing image processing task, and using label to express that the current docker container is operated on the node by Jenkins represented by label;
wherein, the keywords represent processing actions which should be contained in the whole remote sensing image processing flow, and the keywords can contain actions which are contained in a plurality of remote sensing image processing sub-flows; stage represents a description part for executing the remote sensing image processing step, and comprises steps parameters for specifically executing the remote sensing image processing step; steps is an execution command series for executing the remote sensing image processing step in the stage; pipeline represents an entrance mark of the whole remote sensing image processing flow; parallel indicates that the stage contained in it is a parallel execution relation; the job name is added after the build keyword to indicate the task name for starting the remote sensing image processing step.
4. The background Docker task mapping method for the remote sensing image visualization editor of claim 3, wherein the specific process of step S3 is:
s31: designing a process name of a current remote sensing image processing process, generating a new id number of the whole remote sensing processing process according to a timestamp, and judging different remote sensing image processing processes;
s32: representing a remote sensing processing step by using nodes;
s33: representing the execution sequence between the nodes by using connecting lines between the nodes;
s34: one node has N predecessors and N successors, wherein N is more than or equal to 0;
s35: each node mainly contains the following attribute information: the method comprises the steps of processing a remote sensing image, wherein the current node represents a remote sensing image processing sub-process id, a current remote sensing image processing step name, a current remote sensing image processing step id value, a current remote sensing image processing step state, a subsequent remote sensing image processing step id value, a property used for recording a job and running information selected by a user, a machine used for recording an execution machine selected by the user, and a parallel used for recording whether the current node is a parallel node;
if the current node is not the sub-process, the current node is represented by-1, otherwise, a new remote sensing image processing sub-process id value is generated according to the timestamp; and a mark is used for indicating whether the current remote sensing processing step comprises a sub remote sensing image processing flow.
5. The background Docker task mapping method for the remote sensing image visualization editor according to claim 4, wherein the specific process of step S4 is:
s41: in order to enable a user to customize a remote sensing image processing flow, defining that pipeline can add nodes, delete nodes, edit nodes, add connecting lines between nodes, delete connecting lines between nodes, and analyzing a remote sensing processing step to be processed;
s42: and a node is used for representing the sub pipeline, the sub pipeline can be edited by clicking the node, and the content in the sub pipeline can be hidden and expanded.
6. The background Docker task mapping method for the remote sensing image visualization editor according to claim 5, wherein the specific process of step S5 is:
s51: observing the health condition of a Docker engine machine to be deployed by using a node management page of Jenkins, and allocating a label to the current machine, wherein the label is used for identifying which machine is adopted to execute the Docker in a front-end process;
s52: remotely logging in the selected machine, and installing a Docker engine on the slave node to construct a Docker container environment capable of using remote sensing professional processing;
s53: the label value of the current machine is recorded in a database table.
7. The background Docker task mapping method for the remote sensing image visualization editor according to claim 6, wherein the specific process of step S6 is:
s61: calling a timestamp generation method to obtain a function method accurate to millisecond, and taking the function method as an id number of a remote sensing image processing flow;
s62: the front end sends a request to the back end to obtain a jobname and a jobid corresponding to the remote sensing image processing step established in the current Jenkins;
s63: developing a remote sensing service process editor, wherein the editor uses nodes to represent remote sensing image processing steps, can show a job name and a job id corresponding to the remote sensing image processing steps established in the current Jenkins for a user, and is used for selecting the remote sensing image processing steps integrated in the Jenkins for the user;
s64: the remote sensing service process editor can record the remote sensing image processing service process and the service of the attribute value of each node according to the visual customization result of the user, and can record the remote sensing image processing service process and the id number of the remote sensing image processing process so as to form a JSON description file of the remote sensing image processing process example;
s65: and when the user clicks the Save button, sending an http request and sending the JSON file of the front end to the back end.
8. The background Docker task mapping method for the remote sensing image visualization editor of claim 7, wherein the specific process of step S7 is:
s71: the rear end receives a front end http request by using spring @ annotation and receives a JSON file sent by the front end;
s72: extracting nodes, relationships among the nodes and attribute values of the nodes according to a JSON-described remote sensing image processing flow, describing the nodes in the flow by using vertexes in a DAG (direct current) graph, describing the flow relationships among the nodes by using edges in the DAG graph, wherein the attribute values of the vertexes represent the attributes of the nodes;
s73: traversing a DAG graph, and gradually translating a Jenkins task logic control description flow in a Jenkins Pipeline DSL syntax format according to a description method of a remote sensing image processing service flow defined by two steps of keywords agent, sites, stage, steps, Pipeline, parallel and build;
s74: when the JSON file is analyzed and the keyword agent in the member object attribute value in the JSON is found to contain a docker parameter, the fact that the remote sensing image processing step corresponding to the current process needs to be mapped into the docker to be executed is indicated;
s75: when analyzing the Docker grammar mapping, if the user front end selects to download the mirror image from the warehouse, adding alwaysPull statement in Jenkins Pipeline DSL Docker statement, if the user adds the directory mounted between the virtual machine and the Docker at the front end, adding statements args '/mounted directory,/mounted directory ', if the user selects to define the working interval, adding the statement cumStomWorkspace ' user-defined working interval ', and when the user selects to pull the mirror image from the warehouse, adding a statement image ' RemoteSensingProcessing according to the mirror image name input by the user at the front end page, adding a statement label ' nodeX ' according to the label name input by the user at the front end page, and specifying the label of the running machine;
s76: according to the traversal translation result, respectively splicing DSL statements to obtain Jenkins Pipeline DSL statements;
s77: and reading the data table, judging whether the current data table contains the id value of the remote sensing image processing flow, if not, adding a new remote sensing image processing flow record, and if the data table is stored, refreshing the existing data record.
9. The background Docker task mapping method for the remote sensing image visualization editor of claim 8, wherein the specific process of step S8 is:
s81: generating a new jobb by using the analyzed Jenkins Pipeline DSL according to a template for creating a jobb configuration.
S82: taking the generated jobfile in the xml as a parameter, and calling crete jobapi in Jenkins api to create a remote sensing image processing flow task in Jenkins;
s83: after a user triggers a remote sensing image processing flow execution button at the front end, a request is sent to the rear end, and the rear end calls a jenkins api to trigger a jenkins jobcorresponding to the current remote sensing image processing flow to execute a task;
s84: and (3) analyzing the jobconfig. xml file by Jenkins, and distributing the remote sensing processing steps into different Docker containers or Jenkins slave nodes for processing according to parallel and serial processing steps.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910073457.8A CN109857462B (en) | 2019-01-25 | 2019-01-25 | Background Docker task mapping method of remote sensing image visual editor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910073457.8A CN109857462B (en) | 2019-01-25 | 2019-01-25 | Background Docker task mapping method of remote sensing image visual editor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109857462A CN109857462A (en) | 2019-06-07 |
| CN109857462B true CN109857462B (en) | 2021-07-09 |
Family
ID=66896137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910073457.8A Expired - Fee Related CN109857462B (en) | 2019-01-25 | 2019-01-25 | Background Docker task mapping method of remote sensing image visual editor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109857462B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949915A (en) * | 2020-08-18 | 2020-11-17 | 河南大学 | Visual customization method and system for production process of remote sensing product |
| CN114489957B (en) * | 2022-04-01 | 2022-07-22 | 国家卫星海洋应用中心 | Remote sensing satellite data processing method and device and electronic equipment |
| CN117876840A (en) * | 2023-11-30 | 2024-04-12 | 中国科学院空天信息创新研究院 | Remote sensing basic model rapid training method and system based on template editing |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101419623B (en) * | 2008-12-09 | 2010-09-08 | 中山大学 | Geographical simulation optimizing system |
| US10891255B2 (en) * | 2015-03-18 | 2021-01-12 | Intel Corporation | Heterogeneous multiprocessor including scalar and SIMD processors in a ratio defined by execution time and consumed die area |
| US10452552B2 (en) * | 2017-04-17 | 2019-10-22 | Intel Corporation | Memory-based dependency tracking and cache pre-fetch hardware for multi-resolution shading |
| CN107423053B (en) * | 2017-06-15 | 2020-08-21 | 东莞理工学院 | Web model packaging and distributed processing method for remote sensing image processing |
| CN107506892B (en) * | 2017-07-17 | 2018-12-21 | 北京空间飞行器总体设计部 | One kind towards quiet rail Optical remote satellite star integral intelligent control system |
| CN107526573B (en) * | 2017-07-28 | 2020-12-22 | 中国资源卫星应用中心 | Method for processing remote sensing image by adopting parallel pipeline |
| CN108196843B (en) * | 2018-01-09 | 2021-03-26 | 成都睿码科技有限责任公司 | Operation and maintenance method for automatic compilation and deployment of visual Docker container |
| CN108897527B (en) * | 2018-05-21 | 2022-04-15 | 东莞理工学院 | Docker mirror image automatic dynamic construction method for remote sensing image processing in cloud computing |
-
2019
- 2019-01-25 CN CN201910073457.8A patent/CN109857462B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN109857462A (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105700888B (en) | A kind of visualization quick development platform based on jbpm workflow engine | |
| CN108304201B (en) | Object updating method, device and equipment | |
| US8417798B2 (en) | Deploying artifacts for packaged software application in cloud computing environment | |
| CN102521120B (en) | Software automation test system and method | |
| CN109857462B (en) | Background Docker task mapping method of remote sensing image visual editor | |
| CN109375914B (en) | Information remote interaction method and system | |
| WO2018036342A1 (en) | Csar-based template design visualization method and device | |
| CN111930635A (en) | Swagger-based rapid automatic testing method and system | |
| US9720656B2 (en) | Method and apparatus for enabling agile development of services in cloud computing and traditional environments | |
| CN111241454A (en) | Method, system and device for generating webpage code | |
| CA2391719A1 (en) | Editing files of remote systems using an integrated development environment | |
| CN111104103B (en) | Visualization method and system for software editing micro-service | |
| JP2008171435A (en) | Extending attribute of application generated using fourth generation programming tool | |
| CN112764736B (en) | Web end flow chart modeling method, device and system | |
| US20140165038A1 (en) | Analysis of propagated information using annotated forests | |
| CN112199373A (en) | Form development method and system | |
| CN105824616A (en) | Development method and frame for equipment control system program | |
| CN112083926A (en) | Web user interface generation method and device | |
| US8700374B1 (en) | System and method of using an active link in a state programming environment to locate an element in a graphical programming environment | |
| CN116774989A (en) | Design and implementation method, system and terminal of software construction and release device | |
| US20130138690A1 (en) | Automatically identifying reused model artifacts in business process models | |
| JP6981124B2 (en) | System construction support device, system construction support method, and program | |
| CN113778541A (en) | Applet generation method and device, electronic equipment and storage medium | |
| Campos-López et al. | Towards automating the construction of augmented reality interfaces for information systems | |
| CN117251231B (en) | Animation resource processing method, device and system and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210709 Termination date: 20220125 |