CN109086033A - A kind of process dispatch method of Remote Sensing Data Processing - Google Patents
A kind of process dispatch method of Remote Sensing Data Processing Download PDFInfo
- Publication number
- CN109086033A CN109086033A CN201810735060.6A CN201810735060A CN109086033A CN 109086033 A CN109086033 A CN 109086033A CN 201810735060 A CN201810735060 A CN 201810735060A CN 109086033 A CN109086033 A CN 109086033A
- Authority
- CN
- China
- Prior art keywords
- keyword
- module
- input
- nickname
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
Abstract
The invention discloses a kind of process dispatch methods of Remote Sensing Data Processing, its overall step are as follows: one, establish keyword identification library: in new card module registration, analyze its interface document template, input and output keyword is obtained, register it in a manner of nickname the keyword identification library with source module interface uniqueness；Two, flow custom: Keyword List, keyword nickname relationship and flow scheduling module and single-chain type that the main method of calling for completing process, process use call the sequential relationship of all modules；Three, unordered polling dispatching: for the process for only supporting single-chain type to dispatch, using traditional algorithmic dispatching mode；Conversely, using unordered polling dispatching.Flow of task module of the present invention is in unordered state, uses the data ready of each module of poll for module Call Condition, the processing module of data ready is started, can be in the parallel processing for truly realizing module.
Description
Technical field
The present invention relates to a kind of process dispatch method more particularly to a kind of process dispatch methods of Remote Sensing Data Processing.
Background technique
With the development of remote sensing S/M Industry, the image data resolution that the mankind obtain is higher, data volume is bigger, remote sensing shadow
As processing system handles the data of G up to a hundred easily daily, thus satellite image flow chart of data processing requirement of real-time is more reinforced
It is strong.Meanwhile in the state that let a hundred flowers blossom and a hundred schools of thought contend in terms of Remote Sensing Data Processing, but development side is different with difference
Interface form, the module plug-in that existing scheduling scheme is difficult the easy more families of realization is integrated.Existing scheduling system is not for
The module plug-in of same development side is integrated, and mainly there are two schemes: 1) using custom plug-in system form, form special processing stream
Journey is integrated in a manner of development side's internal schedule；Which cannot be truly realized the multiplexing of plug-in unit；It 2) need to the investment progress of the side of development
The carry out interface modification of great effort, to adapt to the dispatch interface requirement of platform；Although this method can form unified tune
Degree mode, but need largely to be exchanged between module development side and scheduling system and module plug-in development side need to carry out greatly
The module plug-in retrofit work of amount, virtually increases the risk of product.
In addition, present parallel scheduling system has multiple computing resource nodes, user is allowed to run multiple processing with the time
Task, and for the scheme of these utilizations of resources use are as follows: in multiple processing tasks, balanced point of the module of different processing tasks
It is fitted on each node, but the calling of each module is still with timing in the same processing task, i.e., previous module is complete
At feedback, it is re-used as the foundation of the latter module calling.The program is reallocated resources to be limited by the intrinsic timing of process, cannot be more
Reasonably system resources in computation is efficiently used, cannot achieve real parallel type processing.
Therefore, the situation unreasonable for above system module schedules needs the process tune to existing Remote Sensing Data Processing
Degree method improves.
Summary of the invention
In order to solve shortcoming present in above-mentioned technology, the present invention provides a kind of process tune of Remote Sensing Data Processing
Degree method.
In order to solve the above technical problems, the technical solution adopted by the present invention is that: a kind of process tune of Remote Sensing Data Processing
Degree method, overall step are as follows:
One, keyword identification library is established:
The keyword identification library with source module interface uniqueness is established, it is made while supporting also known as to export with help document
Function, and key word library addition, deletion and more new tool are provided；In new card module registration, its interface document mould is analyzed
Plate obtains input and output keyword, register it in a manner of nickname keyword identification library；Simultaneously by card module keyword
The addition and deletion of home banking are integrated into during the registration and deletion of card module；
Two, flow custom:
The process mainly complete the method for calling of process, the Keyword List that process uses, keyword nickname relationship and
Flow scheduling module and single-chain type call the sequential relationship of all modules, dispatch it for individual task, realize that one kind can
Parallel scheduling strategy；
Three, unordered polling dispatching:
The input and output Keyword List and process that the process mainly completes the scheduling mode of this process, process uses close
Key word nickname relationship and flow scheduling module；The scheduling process supports traditional single-chain type scheduling and unordered polling dispatching；
When an existing process is executed by selection, whether acquisition flow scheduling mode, decision algorithm only support list first
Chain step scheduling, for the process for only supporting single-chain type to dispatch, using traditional algorithmic dispatching mode；Conversely, using unordered
Polling dispatching；
When corresponding to using unordered polling dispatching, all keywords that process is used are extracted first, load input data, and more
Keyword corresponding to new data, poll calling module judge which module input is ready, generation module input, module tune
With；When some resume module in this process is completed, poll determines other data preparation situations being not carried out again, generates
All module input files of data ready, and start these modules.
Further, the registration of card module are as follows: the input and output keyword for obtaining plug-in unit first will be closed with input and output
The existing duplicate keyword of keyword description content in key character library, if keyword name is inconsistent, new keyword is with
There is the mode of the nickname relationship of keyword to be written, if keyword name is consistent, keeps original keyword, keyword registration number adds
1.Input and output key word library directly added for unduplicated keyword, keyword registers number as 1, finally by module plug-in
All information writing module plugin libraries complete card module registration.
Further, the deletion process of card module are as follows: module plug-in is removed from module plug-in library first, is then obtained
The input and output keyword of plug-in unit is taken, keyword times of registration subtracts 1；The registration of the same nickname of keyword existing in a manner of nickname
Number subtracts 1, and the nickname for being 1 for script times of registration directly deletes the nickname.The number of keyword can be 0, this is closed at this time
Key word only provides the Link role of different nicknames.
Further, during flow custom, not homologous processing module input and output keyword overwhelming majority situation
Under be unmatched, and its actual content is consistent, and under this situation, they have common key in keyword identification library
Word, thus they equally can establish front and back input/output relation.
For complicated third party's module plug-in, by way of nickname keyword with the input and output of existing module plug-in
It is connected fitting, realizes the flexibility of scheduling system.
Flow of task module of the present invention is in unordered state, uses the data ready of each module of poll for module
Call Condition starts the processing module of data ready, can be in the parallel processing for truly realizing module.This
Outside, the present invention passes through the module plug-in input and output key word library safeguarded unity, and the input and output to third party's module plug-in
It is registered in the form of keyword nickname, to achieve the effect that it is not necessary to modify card modules to integrate.
Detailed description of the invention
Fig. 1 is overall step flow chart of the invention.
Fig. 2 is the registration process flow chart of card module.
Fig. 3 is the deletion process flow diagram flow chart of card module.
Fig. 4 is the schematic diagram of flow custom process.
Fig. 5 is the flow chart of unordered polling dispatching.
Specific embodiment
The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
A kind of process dispatch method of Remote Sensing Data Processing shown in FIG. 1, overall step are as follows:
One, keyword identification library is established:
The keyword identification library with source module interface uniqueness is established, it is made while supporting also known as to export with help document
Function, and key word library addition, deletion and more new tool are provided；In new card module registration, its interface document mould is analyzed
Plate obtains input and output keyword, register it in a manner of nickname keyword identification library；Simultaneously by card module keyword
The addition and deletion of home banking are integrated into during the registration and deletion of card module；
The registration process of card module is as shown in Figure 2: the input and output keyword of plug-in unit is obtained first, it will be with input and output
The existing duplicate keyword of keyword description content in key word library, if keyword name is inconsistent, new keyword with
The mode of the nickname relationship of existing keyword is written, if keyword name is consistent, keeps original keyword, keyword registers number
Add 1.Input and output key word library directly added for unduplicated keyword, keyword registers number as 1, finally by module plug-in
All information writing module plugin libraries, complete card module registration.
The deletion process of card module is as shown in Figure 3: first removing module plug-in from module plug-in library, then obtains
The input and output keyword of plug-in unit, keyword times of registration subtract 1；The registration time of the same nickname of keyword existing in a manner of nickname
Number subtracts 1, and the nickname for being 1 for script times of registration directly deletes the nickname.The number of keyword can be 0, this is crucial at this time
Word only provides the Link role of different nicknames.
Two, flow custom:
The process mainly complete the method for calling of process, the Keyword List that process uses, keyword nickname relationship and
Flow scheduling module and single-chain type call the sequential relationship of all modules, dispatch it for individual task, realize that one kind can
Parallel scheduling strategy；Process signal is as shown in Figure 4；
During flow custom, not homologous processing module input and output keyword is to mismatch in most cases
, and its actual content is consistent, under this situation, they have common keyword, thus it in keyword identification library
Equally can establish front and back input/output relation.
Three, unordered polling dispatching:
The input and output Keyword List and process that the process mainly completes the scheduling mode of this process, process uses close
Key word nickname relationship and flow scheduling module；The scheduling process supports traditional single-chain type scheduling and unordered polling dispatching；It crosses
Journey signal is as shown in Figure 5；
When an existing process is executed by selection, whether acquisition flow scheduling mode, decision algorithm only support list first
Chain step scheduling, for the process for only supporting single-chain type to dispatch, using traditional algorithmic dispatching mode；Conversely, using unordered
Polling dispatching；It is defeated with the input of existing module plug-in by way of nickname keyword for complicated third party's module plug-in
It is connected fitting out, realizes the flexibility of scheduling system.
When corresponding to using unordered polling dispatching, all keywords that process is used are extracted first, load input data, and more
Keyword corresponding to new data, poll calling module judge which module input is ready, generation module input, module tune
With；When some resume module in this process is completed, poll determines other data preparation situations being not carried out again, generates
All module input files of data ready, and start these modules.
The present invention is verified on geovis ifactory dispatching platform and is used, and can meet quick process demand.
One aspect of the present invention is managed collectively by the input and output to all registration module plug-in units, to the duplicate word of title
Section, times of registration updates, to different names, but the consistent field of description content, it is handled by the way of nickname, thus real
Showed disparate modules effectively integrates unification；On the other hand the module preparation in different task is obtained by polling schemas,
The timing limitation in scheduling process is broken through, thus more rationally balanced distributing system resource, and then improve processing speed.Cause
This, the present invention is dispatched by individual task, it can be achieved that a kind of scheduling strategy that can be parallel；Multiple tasks are dispatched, Ke Yigeng
For the carry out resource allocation of balanced and reasonable；It, can be by using the side of nickname keyword for complicated third party's module plug-in
Formula is connected fitting with the input and output of existing module plug-in, to realize the flexibility of scheduling system.
Above embodiment is not limitation of the present invention, and the present invention is also not limited to the example above, this technology neck
The variations, modifications, additions or substitutions that the technical staff in domain is made within the scope of technical solution of the present invention, also belong to this hair
Bright protection scope.
Claims (5)
1. a kind of process dispatch method of Remote Sensing Data Processing, it is characterised in that: the overall step of the method are as follows:
One, keyword identification library is established:
The keyword identification library with source module interface uniqueness is established, it is made while supporting also known as to export function with help document
Can, and key word library addition, deletion and more new tool are provided；In new card module registration, its interface document template is analyzed,
Input and output keyword is obtained, register it in a manner of nickname keyword identification library；Simultaneously by card module keyword mark
The addition and deletion for knowing library are integrated into during the registration and deletion of card module；
Two, flow custom:
The process mainly completes the method for calling of process, the Keyword List that process uses, keyword nickname relationship and process
Scheduler module and single-chain type call the sequential relationship of all modules, dispatch it for individual task, realize that one kind can be parallel
Scheduling strategy；
Three, unordered polling dispatching:
The input and output Keyword List and flow keyword that the process mainly completes the scheduling mode of this process, process uses
Nickname relationship and flow scheduling module；The scheduling process supports traditional single-chain type scheduling and unordered polling dispatching；
When an existing process is executed by selection, whether acquisition flow scheduling mode, decision algorithm only support single-chain type first
Flow scheduling, for the process for only supporting single-chain type to dispatch, using traditional algorithmic dispatching mode；Conversely, using unordered poll
Scheduling；
When corresponding to using unordered polling dispatching, all keywords that process is used are extracted first, load input data, and update number
According to corresponding keyword, poll calling module judges which module input is ready, and generation module input, module is called；When
When some resume module in this process is completed, poll determines other data preparation situations being not carried out again, generates
All module input files of data ready, and start these modules.
2. the process dispatch method of Remote Sensing Data Processing according to claim 1, it is characterised in that: the card module
Registration are as follows: first obtain plug-in unit input and output keyword, will in input and output key word library it is existing key word description in
Hold duplicate keyword, if keyword name is inconsistent, new keyword is write in a manner of having the nickname relationship of keyword
Enter, if keyword name is consistent, keeps original keyword, keyword registration number adds 1.Unduplicated keyword is directly added
Add input and output key word library, keyword registers number as 1, complete finally by all information writing module plugin libraries of module plug-in
It is registered at card module.
3. the process dispatch method of Remote Sensing Data Processing according to claim 2, it is characterised in that: the card module
Deletion process are as follows: module plug-in is removed from module plug-in library first, then obtains the input and output keyword of plug-in unit, it is crucial
Word times of registration subtracts 1；The times of registration of the same nickname of keyword existing in a manner of nickname subtracts 1, is 1 for script times of registration
Nickname, directly delete the nickname.The number of keyword can be 0, and the tie that this keyword only provides different nicknames at this time is made
With.
4. the process dispatch method of Remote Sensing Data Processing according to claim 3, it is characterised in that: in flow custom process
In, not homologous processing module input and output keyword is unmatched in most cases, and its actual content is consistent
, under this situation, they have common keyword in keyword identification library, thus they equally can establish front and back input
Output relation.
5. the process dispatch method of Remote Sensing Data Processing according to claim 4, it is characterised in that: for complicated third
Square module plug-in is connected fitting by way of nickname keyword with the input and output of existing module plug-in, realizes scheduling system
Flexibility.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810735060.6A CN109086033B (en) | 2018-07-06 | 2018-07-06 | Flow scheduling method for remote sensing data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810735060.6A CN109086033B (en) | 2018-07-06 | 2018-07-06 | Flow scheduling method for remote sensing data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109086033A true CN109086033A (en) | 2018-12-25 |
| CN109086033B CN109086033B (en) | 2021-08-03 |
Family
ID=64837050
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810735060.6A Active CN109086033B (en) | 2018-07-06 | 2018-07-06 | Flow scheduling method for remote sensing data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109086033B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114461357A (en) * | 2021-12-22 | 2022-05-10 | 中国科学院空天信息创新研究院 | Remote sensing satellite raw data real-time processing flow scheduling engine |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100162277A1 (en) * | 2008-12-22 | 2010-06-24 | Electronics And Telecommunications Research Institute Of Daejeon | Apparatus and method for executing robot application program |
| CN102867067A (en) * | 2012-09-28 | 2013-01-09 | 用友软件股份有限公司 | Device and method for data integration processing for heterogenous system |
| CN104125166A (en) * | 2014-07-31 | 2014-10-29 | 华为技术有限公司 | Queue scheduling method and computing system |
| US20150019686A1 (en) * | 2013-07-12 | 2015-01-15 | Seven Networks, Inc. | Distributed caching systems with configurable extended caching optimization |
| US20150334080A1 (en) * | 2014-05-15 | 2015-11-19 | Reinaldo Tamayo | Systems and methods for a keyword/key phrase url and path replacement and management |
| CN105094984A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Resource scheduling method and system |
| CN105426160A (en) * | 2015-11-10 | 2016-03-23 | 北京时代民芯科技有限公司 | Instruction classified multi-emitting method based on SPRAC V8 instruction set |
| CN105653425A (en) * | 2015-12-28 | 2016-06-08 | 中国民航信息网络股份有限公司 | Complicated event processing engine based monitoring system |
| CN106330545A (en) * | 2016-08-26 | 2017-01-11 | 电子科技大学 | Seismic interpretation system and data transmission scheduling method based on system |
| CN107205016A (en) * | 2017-04-18 | 2017-09-26 | 中国科学院计算技术研究所 | The search method of internet of things equipment |
| CN107430517A (en) * | 2015-05-27 | 2017-12-01 | 谷歌公司 | For the online marketplace for the plug-in unit for strengthening conversational system |
-
2018
- 2018-07-06 CN CN201810735060.6A patent/CN109086033B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100162277A1 (en) * | 2008-12-22 | 2010-06-24 | Electronics And Telecommunications Research Institute Of Daejeon | Apparatus and method for executing robot application program |
| CN102867067A (en) * | 2012-09-28 | 2013-01-09 | 用友软件股份有限公司 | Device and method for data integration processing for heterogenous system |
| US20150019686A1 (en) * | 2013-07-12 | 2015-01-15 | Seven Networks, Inc. | Distributed caching systems with configurable extended caching optimization |
| US20150334080A1 (en) * | 2014-05-15 | 2015-11-19 | Reinaldo Tamayo | Systems and methods for a keyword/key phrase url and path replacement and management |
| CN104125166A (en) * | 2014-07-31 | 2014-10-29 | 华为技术有限公司 | Queue scheduling method and computing system |
| CN105094984A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Resource scheduling method and system |
| CN107430517A (en) * | 2015-05-27 | 2017-12-01 | 谷歌公司 | For the online marketplace for the plug-in unit for strengthening conversational system |
| CN105426160A (en) * | 2015-11-10 | 2016-03-23 | 北京时代民芯科技有限公司 | Instruction classified multi-emitting method based on SPRAC V8 instruction set |
| CN105653425A (en) * | 2015-12-28 | 2016-06-08 | 中国民航信息网络股份有限公司 | Complicated event processing engine based monitoring system |
| CN106330545A (en) * | 2016-08-26 | 2017-01-11 | 电子科技大学 | Seismic interpretation system and data transmission scheduling method based on system |
| CN107205016A (en) * | 2017-04-18 | 2017-09-26 | 中国科学院计算技术研究所 | The search method of internet of things equipment |
Non-Patent Citations (2)
| Title |
|---|
| TOMMI TAKALA ET AL.: ""An Adapter Framework for Keyword-Driven Testing"", 《2009 NINTH INTERNATIONAL CONFERENCE ON QUALITY SOFTWARE》 * |
| 李传荣 等: ""通用遥感数据并行处理框架设计与实现"", 《科学技术与工程》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114461357A (en) * | 2021-12-22 | 2022-05-10 | 中国科学院空天信息创新研究院 | Remote sensing satellite raw data real-time processing flow scheduling engine |
| CN114461357B (en) * | 2021-12-22 | 2022-11-11 | 中国科学院空天信息创新研究院 | Remote sensing satellite original data real-time processing flow scheduling system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109086033B (en) | 2021-08-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102982075B (en) | Support to access the system and method for heterogeneous data source | |
| US8725707B2 (en) | Data continuous SQL process | |
| US4894771A (en) | Data base management system extending structure | |
| US10515118B2 (en) | Processing a data flow graph of a hybrid flow | |
| CN102542007A (en) | Method and system for synchronization of relational databases | |
| CN102033748A (en) | Method for generating data processing flow codes | |
| CN107193643B (en) | A kind of highly reliable distributed traffic real-time statistical method and system | |
| CN103942281B (en) | The method and device that a kind of object to persistent storage is operated | |
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| CN101382893B (en) | On-line assembling method for component based on Web service | |
| WO2019047441A1 (en) | Communication optimization method and system | |
| CN105930417B (en) | A kind of big data ETL interactive process platform based on cloud computing | |
| CN104618433A (en) | ECA (Event-Condition-Action) rule based service combination method | |
| CN106339498A (en) | Method, device and system for synchronizing data | |
| CN109902117A (en) | Operation system analysis method and device | |
| CN108073688A (en) | A kind of method and device of Data Migration | |
| CN109241104A (en) | The resolver and its implementation of AISQL in decision type distributed data base system | |
| CN103399735A (en) | Method for developing intermediate layer of remote function call interface | |
| CN100407663C (en) | Universal testing system and method for telecommunication intelligent service | |
| CN106445645A (en) | Method and device for executing distributed computation tasks | |
| CN103164476A (en) | Execution method and execution device of applying metadata to describe files in business intelligence (BI) | |
| O'Brien et al. | SEEK: Accomplishing enterprise information integration across heterogeneous sources. | |
| US20220138195A1 (en) | User defined functions for database query languages based on call-back functions | |
| CN108345658A (en) | Algorithm calculates decomposing process, server and the storage medium of track | |
| CN108255467A (en) | The method, apparatus and system of service are performed according to workflow sequence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant after: Zhongke Star Map Co., Ltd. Address before: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant before: Space Star Technology (Beijing) Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Wu Fangcai Inventor after: Zhang Difei Inventor after: Bai Xiaohui Inventor after: He Xiaoning Inventor after: Zhou Xin Inventor before: Wu Fangcai Inventor before: Bai Xiaohui Inventor before: He Xiaoning Inventor before: Zhou Xin |