CN105426235A - Depicting method for terrestrial atmospheric aerosol retrieval distributed workflow dependency - Google Patents
Depicting method for terrestrial atmospheric aerosol retrieval distributed workflow dependency Download PDFInfo
- Publication number
- CN105426235A CN105426235A CN201510747043.0A CN201510747043A CN105426235A CN 105426235 A CN105426235 A CN 105426235A CN 201510747043 A CN201510747043 A CN 201510747043A CN 105426235 A CN105426235 A CN 105426235A
- Authority
- CN
- China
- Prior art keywords
- data
- gasoloid
- node
- task
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a method for depicting terrestrial atmospheric aerosol retrieval distributed workflow dependency, and relates to the field of high-performance geocomputation, in particular to cloud computing-oriented MOSDIS (Moderate-Resolution Imaging Spectroradiometer) data. According to the method, through establishment of a multiple element group, six key elements are used to describe a finite graph distributed in a network in a mathematical formalization manner to realize mathematical formalization depiction of the aerosol remote sensing retrieval distributed workflow dependency; and after related remote sensing processing tasks, configuration and control information are acquired through interpretation of the abstract depicting method, an abstract service flow is subjected to resource mapping to obtain Web Service-encapsulated remote sensing retrieval processing and data processing related service to drive an aerosol retrieval processing process to be executed in the distributed calculation condition. In a flow definition phase, the Web Services are called; and through conversion control of four construction modules of AND-split, AND-join, OR-split and OR-join, ordered execution can be realized according to the relations between parallel and serial, and predecessor and successor, so that the quick retrieval of aerosol is realized.
Description
Technical field
The present invention relates to high-performance geoscience computing field, the MODIS data land atmospheric aerosol inverting distributed work flow dependence depicting method of especially facing cloud calculating.
Background technology
The abbreviation of the English moderate-resolutionimagingspectroradiometer of MODIS(, full name is Moderate Imaging Spectroradiomete) data land Atmospheric Aerosol Retrieval with Remote Sensing Technique needs to carry out the calculating of a large amount of data prediction and complicated radiation transfer equation, operand is large, length consuming time.These processing procedures mainly comprise, the data prediction such as calibration, radiation transmission, set correction, the image procossing such as band math, image conversion, data fusion, the operations such as Solving Nonlinear Systems of Equations isoinversion computing and aftertreatment.Adopt the Distributed Workflow Technology towards the atmospheric aerosol inverting of MOSDIS data land can decompose these process steps, form the subtask having dependence each other, and run in distributed computer environment according to subtask precursor and follow-up, parallel and Serial Relation, and then the overall inverting processing time can be reduced, thus realize aerocolloidal fast inversion.When being described these processes, adopt the mode such as natural language or data flow diagram, can cause conflicting statement and the understanding to statement different modes, almost invariably there will be ambiquity, and these describing modes be imperfect, abstraction hierarchy is chaotic.Therefore need to adopt the method for mathematical form to carry out succinctly describing exactly the result of its state, phenomenon or action to said process, realize seamlessly transitting between different soft project activities, but the method how do not realized in prior art, how to realize is the technical matters that those skilled in the art constantly explore.
Summary of the invention
The object of the invention is to overcome the deficiencies in the prior art, a kind of land atmospheric aerosol inverting distributed work flow dependence depicting method is provided, realize performing in order with serial, precursor and successor relationship according to parallel in distributed computer environment, realize aerocolloidal fast inversion.
In order to achieve the above object, the technical solution used in the present invention is:
A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method, described method, towards MODIS data, comprises the steps:
Step 1. according to finite graph DND=(S, T, I, O, N, D) rule build remote-sensing inversion service flow describer abstract description is carried out to gasoloid quantitative inversion treatment step, be formed with the abstract service stream of dependence;
Step 2. service flow resolver is resolved according to task, configuration information, the several classification of control information abstract service stream, is decomposed into the discernible correlation parameter of Service Matching device, abstract service stream is carried out resource mapping;
Step 3. Service Matching device, according to obtaining of task, match information, control information, manage with UDDI service register center, metadata catalog, deblocking, task manager communicates, the operation of scheduling correlation module;
Step 4. task manager is according to the address of service obtained, data block address, control information, configuration information, actuate actuators according to the order of sequence, parallel/serial row perform Web service stream (i.e. web service stream), and to implementation monitoring, report an error to abnormal, and restart execution;
When step 5. actuator runs into gasoloid inversion procedure service call point in the process of implementation, in service registry storehouse, find corresponding Web service send to actuator, carry out example execution, and obtain result;
In the process of implementation, actuator takes out the service of gasoloid inversion procedure and binds and call step 6. successively from service registry storehouse; When the failure of gasoloid inversion procedure service call, it is deleted from service registry storehouse, and trigger exception handler and process;
Step 7. repeats step 3-6, until flow performing is to without follow-up point of invocation, completes the inverting of atmospheric aerosol;
Described DND rule wants unit to describe a finite graph be distributed on network by six, and DND=(S, T, I, O, N, D); Wherein S={s
1, s
2..., s
pthe set of gasoloid remote-sensing inversion Processing Algorithm state Finite is described, p>0 is the number of S collection; T={t
1, t
2..., t
kthis finite aggregate of transfer process is described, k>0 is the number of transfer process; N={n
1, n
2..., n
qbe network data node IP address collection, q>0 is back end number in network; D={d
1, d
2..., d
ube data set and point cut set thereof, u>0 is a MODISHDF(HDF is the abbreviation of HierarchicalDataFormat, can store dissimilar image and the file layout of digital data) the data block number of formatted file and segmentation thereof; I:D × S × T->Z is input function, define HDFS(and the Hadoop distributed file system be distributed on different pieces of information node) repeat number of data file at the directed arc of S to T or the set of power, Z={0 herein, 1 ... be set of nonnegative integer; O:T × D × S->N is output function, defines gasoloid remote-sensing inversion Processing Algorithm and changes on different pieces of information node, and result is remapped to the set of the process of heterogeneous networks.
As preferred embodiment of the present invention, S={s of the present invention
1, s
2..., s
pin gasoloid remote sensing quantitative Distributed Services stream running status is portrayed, comprise following treatment step:
Step S101. pair warp and weft degree, solar zenith angle, sensor zenith angle wave band carry out the extraction of MODIS data;
Step S102. according to 10KM, 1KM, 500M, carries out longitude and latitude and the resampling of zenith angle data to original MODIS1KM resolution data;
Step S103., according to survey region, carries out the cutting of MODIS image according to longitude and latitude to zenith angle;
Step S104. carries out MODIS image joint according to solar zenith angle, sensor zenith angle;
Step S105. carries out geometry correction according to longitude and latitude to solar zenith angle, sensor zenith angle;
Step S106. extracts cloud mask information from MODIS source document;
Step S107. from original MODIS extracting data Band1, the performance reflectivity of 2,3,4,7;
Step S108., according to ground observation station longitude and latitude and satellite transit time, extracts the ground station observation data with satellite synchronization;
Step S109. calculates the abbreviation of the English AdvancedOxidationTechnologies of AOT(, and the Chinese meaning is high-level oxidation technology) initial value of iterative inversion;
Step S110. calculates AOT;
Step S111. surveys with ground and compares, checking inversion result.
As preferred embodiment of the present invention, D={d of the present invention
1, d
2..., d
ucomprise following data and point cut set:
Step S201. is by level one data product MOD02, MOD03 data of Moderate Imaging Spectroradiomete, air standard product MOD35, MOD04 data, AERNET(AErosolROboticNETwork) these pending data sources of data are incorporated to same catalogue according to chronomere sky, then according to the moon, year forming higher level data directory, input data set is formed with this;
The intermediate processing results of step S202. pair of hdr and the img form produced with merging or the remote sensing special software after merging carries out filing and processes, and task management set of node obtains archival one from T, filing data collection in the middle of being formed;
Intermediate treatment after step S203. pair and bifurcated or bifurcated inputs data hdr, img and carries out deblocking, and from the idle node that T concentrates task management set of node to obtain N to concentrate, by data-mapping to these idle nodes, formation deblocking collection;
Step S203. builds metadata management collection to middle filing data collection and deblocking collection;
The whole flow processing result of step S204. forms output processing result set.
As preferred embodiment of the present invention, S collection of the present invention comprises the process of ground station observation data, information extraction, resampling, cutting, geometry correction, splicing, cloud information extraction, wave band extraction, cloud mask, AOT inverting initial value calculates, AOT calculates and comparatively validate step is surveyed on ground, described step is by IDL(interactive data language InteractiveDataLanguage), Matlab, C++ develop, then compiling is encapsulated into web services form.
As preferred embodiment of the present invention, T collection of the present invention comprises the gasoloid remote-sensing inversion Processing tasks collection based on dot-product operation, field computing, global operations, the distributed access of remote sensing source data, deblocking method, data compression method, data index method, gasoloid remotely-sensed data metadata management, registration, service discovery, the task management of gasoloid remote-sensing inversion, mapping, the monitoring of gasoloid remote sensing data transmission engine, node state, node tasks are monitored, node tasks configures, node tasks is restarted, the polymerization of gasoloid remote-sensing inversion result.
As preferred embodiment of the present invention, D of the present invention integrates as MOSDISHDF data set and point cut set thereof, comprises gasoloid remote sensing source data, MOSDISHDF formatted file and partitioned data set thereof under distributed environment, gasoloid remote sensing intermediate processing results collection, gasoloid remote sensing result collection, gasoloid remote sensing metadata set.
As preferred embodiment of the present invention, N of the present invention integrates as distributed network node management collection and data section point set, comprises Resource Management node collection, task scheduling set of node, Mission Monitor set of node, tasks carrying set of node, data section point set.
As preferred embodiment of the present invention, I of the present invention to integrate as user program according to task depicting language, performs set of scripts, task management command set and submit a task to, and the information of task is sent to the gasoloid remote-sensing inversion task management set of node in the set of T transfer process; Gasoloid remote-sensing inversion task management set of node is the core of whole gasoloid remote-sensing inversion duty mapping, the data set needing D to concentrate with and point cut set machine timed communication, gasoloid remote sensing program in management S should be run on which the corresponding machine in N respectively, needs operations such as managing mission failures all in N, restart; It is the part in whole distributed system there being tasks carrying node that T interior joint Mission Monitor and node condition are monitored, and T interior joint Mission Monitor and node condition monitor the resource situation of implementation status that the thing done is monitor task and oneself place machine; The running status that T interior joint Mission Monitor obtains can send to the gasoloid remote-sensing inversion task management set of node in T by heartbeat; Gasoloid remote-sensing inversion task management set of node can collect above information to operate on which the corresponding machine in N to the new task matching submitted to; The repeat number of HDFS data file at the directed arc of S to T or the set of power in D set, control to have between service flow between the subtask of dependence precursor and follow-up, to walk abreast and Serial Relation runs and performs sequence in distributed computer environment.
As preferred embodiment of the present invention, O collection of the present invention is that in management S set, gasoloid remote-sensing inversion Processing Algorithm calculates in difference and back end maps and the management process truly performed, gasoloid remote-sensing inversion task management in T carries out task ranking to each application that user submits to, to be left unused information by the computational resource obtained from gasoloid remotely-sensed data metadata management, task in scheduler task sequence distributes to inner task further, and carry out node tasks configuration, starter node Mission Monitor, and again restart for task application resource carries out node tasks when task run is failed, after node tasks execution terminates, according to the HDFS data file obtained from gasoloid remotely-sensed data metadata management/deblocking index mark information, after sub-thread computes in N in back end oneself data complete, the gasoloid remote-sensing inversion result polymerization main thread antithetical phrase thread computes result in T carries out remapping based on the data of N data section point set and being polymerized.
As preferred embodiment of the present invention, the abstract service stream described in step 1 of the present invention, 2, service flow are resolved, resource mapping process concrete steps are as follows:
Step S301. provides correlation parameter information to carry out validation verification to abstract service stream according to service flow;
Step S302. matches according to model bank to the task in abstract stream;
Step S303. produces running environment configuration information to the task after pairing according to parameter that service flow provides;
Step S304. parses the Task Dependent relation after pairing;
Physical resource mapping is carried out after step S305. service flow successfully resolved.
Compared with prior art, the invention has the beneficial effects as follows:
The first, in concrete practice of software, treatment scheme in distributed work flow adopts WebService technology to encapsulate, the details of these WebServices are comprised in flow definition, as type, address etc., by the general standard interface that it builds, each species diversity of heterogeneous platform in shielding network;
The second, the workflow processing task of gasoloid remote-sensing inversion maps according to polynary group of DND mathematical form describing mode under distributed network environment, in the flow definition stage, the WebServices called can be the WebServices that each XM is developed separately, also can be the function interface that each engine WebServices externally provides;
Three, mathematical form description and flow definition form the subtask having dependence each other, subtask between distributed network by service call agreement SOAP, Service Description Protocol WSDL and service discovery/integrated UDDI, and service discovery, the upper layer application such as integrated;
Four, by bifurcated, with merge or bifurcated or merges four kinds of constructing modules and carry out circulation and control, can to realize in distributed computer environment, according to walk abreast and serial, precursor and successor relationship perform in order, realizing aerocolloidal fast inversion.
Accompanying drawing explanation
Fig. 1 is the system architecture method flow diagram of gasoloid remote sensing of the present invention quantitative Distributed Services stream running status;
Fig. 2 is that abstract service stream of the present invention is resolved and mapping process figure.
Embodiment
Purport of the present invention is to overcome the deficiencies in the prior art, a kind of mathematical form depicting method towards MOSDIS data land Atmospheric Aerosol Retrieval with Remote Sensing Technique distributed work flow dependence is provided, describes the dependence in inversion procedure process, parallel and Serial Relation.By this mathematical form depicting method, related scheduling algorithm can be write and drive the orderly execution of gasoloid inversion procedure process under distributed computing environment.Be described in detail, to be interpretated more in-depth technical characteristic of the present invention and advantage with reference to accompanying drawing below in conjunction with embodiment.
As shown in Figure 1, the present invention sets up software architecture method to method flow diagram of the present invention on mathematical form basis, is one of development computer control system strict and effective approach.DND=(S, T, I, O, N, D) mode of polynary group of different forms describes the configuration definition of assembly, connector and data in software architecture, and restriction relation between these elements and attribute thereof, and its adopts the behavior in each stage of mode descriptive system and the migration of state of mathematical form.Many first group of DND can carry out formalization to the system architecture of the gasoloid remote sensing quantitative Distributed Services stream running status shown in Fig. 1 and portrays.The process of description shown in Fig. 1 is as follows:
The first step, carries out abstract description according to the remote-sensing inversion service flow describer that DND rule builds to gasoloid quantitative inversion treatment step, is formed with the abstract service stream of dependence;
Second step, service flow resolver is resolved according to task, configuration information, the several classification of control information abstract service stream, is decomposed into the discernible correlation parameter of Service Matching device;
3rd step, Service Matching device, according to obtaining of task, match information, control information, manage with UDDI service register center, metadata catalog, deblocking, task manager communicates, the operation of scheduling correlation module;
4th step, task manager is according to the address of service obtained, data block address, control information, configuration information, and actuate actuators is according to the order of sequence, parallel/serial row performs Web service stream, and to implementation monitoring, report an error, and restart execution to abnormal;
5th step, when actuator runs into gasoloid inversion procedure service call point in the process of implementation, finds corresponding web services and sends to actuator, carry out example execution, and obtain result in service registry storehouse;
6th step, in the process of implementation, actuator takes out the service of gasoloid inversion procedure successively and binds and call from service registry storehouse; When the failure of gasoloid inversion procedure service call, it is deleted from service registry storehouse, and trigger exception handler and process;
7th step, repeat third and fourth, five, six steps, until flow performing is to not having follow-up point of invocation.
The present invention, from mathematical form, wants unit to describe a finite graph be distributed on network, DND=(S, T, I, O, N, D with 6).Wherein S={s
1, s
2..., s
pthe set of gasoloid remote-sensing inversion Processing Algorithm state Finite is described, p>0 is S collection number; T={t
1, t
2..., t
kthis finite aggregate of transfer process is described, k>0 is the number of transfer process; N={n
1, n
2..., n
qbe network data node IP address collection, q>0 is back end number in network; D={d
1, d
2..., d
ube data set and point cut set thereof, u>0 is the data block number of a MOSDISHDF formatted file and segmentation thereof; I:D × S × T->Z is input function, it defines and is distributed on the repeat number of HDFS data file at the directed arc of S to T on different pieces of information node or the set of power, here Z={0,1 ... } be set of nonnegative integer; O:T × D × S->N is output function, it defines gasoloid remote-sensing inversion Processing Algorithm and changes on different pieces of information node, and result is remapped to the set of the process of heterogeneous networks.
Preferably, S={s of the present invention
1, s
2..., s
pin gasoloid remote sensing quantitative Distributed Services stream running status is portrayed, mainly comprise the process of ground station observation data, information extraction, resampling, cutting, geometry correction, splicing, cloud information extraction, wave band extraction, cloud mask, AOT inverting initial value calculates, AOT calculates and the steps such as comparatively validate are surveyed on ground.These treatment steps mainly use IDL, Matlab, C++ to develop, and then compiling is encapsulated into web services form.
Preferably, T={t of the present invention
1, t
2..., t
kin transfer process set description, mainly comprise the gasoloid remote-sensing inversion Processing tasks collection based on dot-product operation, field computing, global operations, the distributed access of remote sensing source data, deblocking method, data compression method, data index method, gasoloid remotely-sensed data metadata management, registration, service discovery, the task management of gasoloid remote-sensing inversion, mapping, the monitoring of gasoloid remote sensing data transmission engine, node state, node tasks monitoring, node tasks configuration, node tasks restarts, gasoloid remote-sensing inversion result polymerization etc.
Preferably, D={d of the present invention
1, d
2..., d
umainly MOSDISHDF data set and point cut set, mainly gasoloid remote sensing source data, MOSDISHDF formatted file and partitioned data set thereof, gasoloid remote sensing intermediate processing results collection, gasoloid remote sensing result collection, gasoloid remote sensing metadata set under distributed environment.
Preferably, N={n of the present invention
1, n
2..., n
qbe distributed network node management collection and data section point set, comprise Resource Management node collection, task scheduling set of node, Mission Monitor set of node, tasks carrying set of node, data section point set.
Preferably, I:D × S of the present invention × T->Z is that user program is according to task depicting language, perform set of scripts, task management command sets etc. submit a task to, the information of task can be sent to the gasoloid remote-sensing inversion task management set of node in the set of T transfer process, gasoloid remote-sensing inversion task management set of node is the core of whole gasoloid remote-sensing inversion duty mapping, the data set that it needs D to concentrate with and point cut set machine timed communication, gasoloid remote sensing program in management S should be run on which machine in N respectively, need to manage mission failures all in N, the operation such as to restart.It is the part in whole distributed system there being tasks carrying node that T interior joint Mission Monitor and node condition are monitored, the thing mainly implementation status of monitor task and the resource situation of oneself place machine that it does.The running status that T interior joint Mission Monitor obtains can send to the gasoloid remote-sensing inversion task management set of node in T by heartbeat.Gasoloid remote-sensing inversion task management set of node can collect these information to operate on which machine in N to the new task matching submitted to.The repeat number of HDFS data file at the directed arc of S to T or the set of power in D set, control to have between service flow between the subtask of dependence precursor and follow-up, to walk abreast and Serial Relation runs and performs sequence in distributed computer environment.
Preferably, O:T × D of the present invention × S->N is that in management S set, gasoloid remote-sensing inversion Processing Algorithm calculates in difference and back end maps and the management process truly performed, gasoloid remote-sensing inversion task management in T carries out task ranking to each application that user submits to, to be left unused information by the computational resource obtained from gasoloid remotely-sensed data metadata management, task in scheduler task sequence distributes to inner task further, and carry out node tasks configuration, starter node Mission Monitor, and again restart for task application resource carries out node tasks when task run is failed.After node tasks execution terminates, according to the HDFS data file obtained from gasoloid remotely-sensed data metadata management/deblocking index mark information, after sub-thread computes in N in back end oneself data complete, the gasoloid remote-sensing inversion result polymerization main thread antithetical phrase thread computes result in T carries out remapping based on the data of N data section point set and being polymerized.
In addition, S={s of the present invention
1, s
2..., s
pin gasoloid remote sensing quantitative Distributed Services stream running status is portrayed, mainly comprise following treatment step:
The wave bands such as step S101, pair warp and weft degree, solar zenith angle, sensor zenith angle carry out the extraction of MODIS data;
Step S102, to original MODIS1KM resolution data according to 10KM, 1KM, 500M, carry out longitude and latitude and the resampling of zenith angle data;
Step S103, according to survey region, according to longitude and latitude, the cutting of MODIS image is carried out to zenith angle;
Step S104, carry out MODIS image joint according to solar zenith angle, sensor zenith angle;
Step S105, according to longitude and latitude, geometry correction is carried out to solar zenith angle, sensor zenith angle;
Step S106, extract cloud mask information from MODIS source document;
Step S107, from original MODIS extracting data Band1, the performance reflectivity of 2,3,4,7;
Step S108, according to ground observation station longitude and latitude and satellite transit time, to extract and the ground station observation data of satellite synchronization;
The initial value of step S109, calculating AOT iterative inversion;
Step S110, calculating AOT;
Step S111, to survey with ground and compare, checking inversion result.
In order to reach good efficiency of inverse process, D={d of the present invention
1, d
2..., d
umainly contain following data and point cut set:
Step S201, pending to MOD02, MOD03, MOD35, MOD04, AERNET etc. data source being incorporated to same catalogue according to chronomere sky, then according to the moon, year forming higher level data directory, forming input data set with this;
Step S202, to merge or intermediate processing results hdr, img after merging carries out filing and process, and from T task management set of node acquisition archival one, formation centre filing data collection;
Step S203, to bifurcated or bifurcated after intermediate treatment input data hdr, img and carry out deblocking, and task management set of node obtains the idle node in N from T, by data-mapping to these idle nodes, formation deblocking collection;
Step S204, metadata management collection is built to middle filing data collection and deblocking collection;
Step S205, whole flow processing result form output processing result set.
Abstract service stream parsing of the present invention and mapping process figure as shown in Figure 2, describe abstract service stream, service flow are resolved, resource mapping process, specific as follows:
Step S301, correlation parameter information is provided to carry out validation verification to abstract service stream according to service flow;
Step S302, the task in abstract stream to be matched according to model bank;
Step S303, to the task after pairing according to parameter that service flow provides generation running environment configuration information;
Step S304, parse the Task Dependent relation after pairing;
Physical resource mapping is carried out after step S305, service flow successfully resolved.
Compared with prior art, the present invention is by structure one polynary group, unit is wanted to describe a finite graph be distributed on network from mathematical form with 6, carry out mathematical form with this to gasoloid remote-sensing inversion distributed work flow dependence to portray, comprise the dependence described in inversion procedure process, parallel and Serial Relation.By the decipher to this abstract depicting method, after obtaining relevant remote sensing Processing tasks, configuration, control information, abstract service flow is carried out resource mapping, obtain the remote-sensing inversion process after WebService encapsulation and data processing related service, drive the execution of gasoloid inversion procedure process under distributed computing environment.In the flow definition stage, the WebServices called can be the WebServices that each XM is developed separately, also can be the function interface that each engine WebServices externally provides.By to bifurcated, with merge or bifurcated or merges four kinds of constructing modules and carry out circulation and control, can to realize in distributed computer environment, according to walk abreast and serial, precursor and successor relationship perform in order, realizing aerocolloidal fast inversion.
Carry out clear, complete description by the technical scheme in above embodiment to the present invention, obviously described embodiment is the embodiment of a part of the present invention, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Claims (10)
1. a land atmospheric aerosol inverting distributed work flow dependence depicting method, described method, towards MODIS data, comprises the steps:
The remote-sensing inversion service flow describer that step 1. builds according to DND rule carries out abstract description to gasoloid quantitative inversion treatment step, is formed with the abstract service stream of dependence;
Step 2. service flow resolver is resolved according to task, configuration information, the several classification of control information abstract service stream, is decomposed into the discernible correlation parameter of Service Matching device, abstract service stream is carried out resource mapping;
Step 3. Service Matching device, according to obtaining of task, match information, control information, manage with UDDI service register center, metadata catalog, deblocking, task manager communicates, the operation of scheduling correlation module;
Step 4. task manager is according to the address of service obtained, data block address, control information, configuration information, and actuate actuators according to the order of sequence, parallel/serial row performs Web service stream, and to implementation monitoring, report an error, and restart execution to abnormal;
When step 5. actuator runs into gasoloid inversion procedure service call point in the process of implementation, in service registry storehouse, find corresponding Web service send to actuator, carry out example execution, and obtain result;
In the process of implementation, actuator takes out the service of gasoloid inversion procedure and binds and call step 6. successively from service registry storehouse; When the failure of gasoloid inversion procedure service call, it is deleted from service registry storehouse, and trigger exception handler and process;
Step 7. repeats step 3-6, until flow performing is to without follow-up point of invocation, completes the inverting of atmospheric aerosol;
Described DND rule wants unit to describe a finite graph be distributed on network by six, and DND=(S, T, I, O, N, D); Wherein S={s
1, s
2..., s
pthe set of gasoloid remote-sensing inversion Processing Algorithm state Finite is described, p>0 is S collection number; T={t
1, t
2..., t
kthis finite aggregate of transfer process is described, k>0 is the number of transfer process; N={n
1, n
2..., n
qbe network data node IP address collection, q>0 is back end number in network; D={d
1, d
2..., d
ube data set and point cut set thereof, u>0 is the data block number of a MODISHDF formatted file and segmentation thereof; I:D × S × T->Z is input function, defines and is distributed on the repeat number of HDFS data file at the directed arc of S to T on different pieces of information node or the set of power, herein Z={0,1 ... } be set of nonnegative integer; O:T × D × S->N is output function, defines gasoloid remote-sensing inversion Processing Algorithm and changes on different pieces of information node, and result is remapped to the set of the process of heterogeneous networks.
2. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 1, is characterized in that, described S={s
1, s
2..., s
pin gasoloid remote sensing quantitative Distributed Services stream running status is portrayed, comprise following treatment step:
Step S101. pair warp and weft degree, solar zenith angle, sensor zenith angle wave band carry out the extraction of MODIS data;
Step S102. according to 10KM, 1KM, 500M, carries out longitude and latitude and the resampling of zenith angle data to original MODIS1KM resolution data;
Step S103., according to survey region, carries out the cutting of MODIS image according to longitude and latitude to zenith angle;
Step S104. carries out MODIS image joint according to solar zenith angle, sensor zenith angle;
Step S105. carries out geometry correction according to longitude and latitude to solar zenith angle, sensor zenith angle;
Step S106. extracts cloud mask information from MODIS source document;
Step S107. from original MODIS extracting data Band1, the performance reflectivity of 2,3,4,7;
Step S108., according to ground observation station longitude and latitude and satellite transit time, extracts the ground station observation data with satellite synchronization;
Step S109. calculates the initial value of AOT iterative inversion;
Step S110. calculates AOT;
Step S111. surveys with ground and compares, checking inversion result.
3. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 2, is characterized in that, described D={d
1, d
2..., d
ucomprise following data and point cut set:
Pending to MOD02, MOD03, MOD35, MOD04, AERNET etc. data source is incorporated to same catalogue according to chronomere sky by step S201., then according to the moon, year forming higher level data directory, forms input data set with this;
Step S202. pair is carried out filing with merging or intermediate processing results hdr, img after merging and processes, and from T, task management set of node obtains archival one, filing data collection in the middle of being formed;
Intermediate treatment after step S203. pair and bifurcated or bifurcated inputs data hdr, img and carries out deblocking, and from the idle node that T concentrates task management set of node to obtain N to concentrate, by data-mapping to these idle nodes, formation deblocking collection;
Step S203. builds metadata management collection to middle filing data collection and deblocking collection;
The whole flow processing result of step S204. forms output processing result set.
4. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 3, it is characterized in that: described S collection comprises the process of ground station observation data, information extraction, resampling, cutting, geometry correction, splicing, cloud information extraction, wave band extraction, cloud mask, AOT inverting initial value calculates, AOT calculates and comparatively validate step is surveyed on ground, described step is developed by IDL, Matlab, C++, and then compiling is encapsulated into web services form.
5. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 4, it is characterized in that: described T collection comprises based on dot-product operation, field computing, the gasoloid remote-sensing inversion Processing tasks collection of global operations, the distributed access of remote sensing source data, deblocking method, data compression method, data index method, gasoloid remotely-sensed data metadata management, registration, service discovery, the task management of gasoloid remote-sensing inversion, map, gasoloid remote sensing data transmission engine, node state is monitored, node tasks is monitored, node tasks configures, node tasks is restarted, gasoloid remote-sensing inversion result is polymerized.
6. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 5, it is characterized in that: described D integrates as MOSDISHDF data set and point cut set thereof, comprise gasoloid remote sensing source data, MOSDISHDF formatted file and partitioned data set thereof under distributed environment, gasoloid remote sensing intermediate processing results collection, gasoloid remote sensing result collection, gasoloid remote sensing metadata set.
7. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 6, it is characterized in that: described N integrates as distributed network node management collection and data section point set, comprises Resource Management node collection, task scheduling set of node, Mission Monitor set of node, tasks carrying set of node, data section point set.
8. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 7, it is characterized in that: described I to integrate as user program according to task depicting language, performs set of scripts, task management command set and submit a task to, and the information of task is sent to the gasoloid remote-sensing inversion task management set of node in the set of T transfer process; Gasoloid remote-sensing inversion task management set of node is the core of whole gasoloid remote-sensing inversion duty mapping, the data set needing D to concentrate with and point cut set machine timed communication, gasoloid remote sensing program in management S should be run on which the corresponding machine in N respectively, needs operations such as managing mission failures all in N, restart; It is the part in whole distributed system there being tasks carrying node that T interior joint Mission Monitor and node condition are monitored, and T interior joint Mission Monitor and node condition monitor the resource situation of implementation status that the thing done is monitor task and oneself place machine; The running status that T interior joint Mission Monitor obtains can send to the gasoloid remote-sensing inversion task management set of node in T by heartbeat; Gasoloid remote-sensing inversion task management set of node can collect above information to operate on which the corresponding machine in N to the new task matching submitted to; The repeat number of HDFS data file at the directed arc of S to T or the set of power in D set, control to have between service flow between the subtask of dependence precursor and follow-up, to walk abreast and Serial Relation runs and performs sequence in distributed computer environment.
9. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 8, it is characterized in that: described O collection is that in management S set, gasoloid remote-sensing inversion Processing Algorithm calculates in difference and back end maps and the management process truly performed, gasoloid remote-sensing inversion task management in T carries out task ranking to each application that user submits to, to be left unused information by the computational resource obtained from gasoloid remotely-sensed data metadata management, task in scheduler task sequence distributes to inner task further, and carry out node tasks configuration, starter node Mission Monitor, and again restart for task application resource carries out node tasks when task run is failed, after node tasks execution terminates, according to the HDFS data file obtained from gasoloid remotely-sensed data metadata management/deblocking index mark information, after sub-thread computes in N in back end oneself data complete, the gasoloid remote-sensing inversion result polymerization main thread antithetical phrase thread computes result in T carries out remapping based on the data of N data section point set and being polymerized.
10. the land atmospheric aerosol inverting distributed work flow dependence depicting method according to any one of claim 1-9, is characterized in that: the abstract service stream described in step 1,2, service flow are resolved, resource mapping process concrete steps are as follows:
Step S301. provides correlation parameter information to carry out validation verification to abstract service stream according to service flow;
Step S302. matches according to model bank to the task in abstract stream;
Step S303. produces running environment configuration information to the task after pairing according to parameter that service flow provides;
Step S304. parses the Task Dependent relation after pairing;
Physical resource mapping is carried out after step S305. service flow successfully resolved.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510747043.0A CN105426235B (en) | 2015-11-06 | 2015-11-06 | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510747043.0A CN105426235B (en) | 2015-11-06 | 2015-11-06 | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105426235A true CN105426235A (en) | 2016-03-23 |
| CN105426235B CN105426235B (en) | 2018-09-25 |
Family
ID=55504461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510747043.0A Expired - Fee Related CN105426235B (en) | 2015-11-06 | 2015-11-06 | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105426235B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6909815B2 (en) * | 2003-01-31 | 2005-06-21 | Spectral Sciences, Inc. | Method for performing automated in-scene based atmospheric compensation for multi-and hyperspectral imaging sensors in the solar reflective spectral region |
| CN102778675A (en) * | 2012-04-28 | 2012-11-14 | 中国测绘科学研究院 | Atmospheric correction method and atmospheric correction module for satellite remote-sensing image |
| CN103020325A (en) * | 2013-01-17 | 2013-04-03 | 中国科学院计算机网络信息中心 | Distributed remote sensing data organization query method based on NoSQL database |
| CN103198097A (en) * | 2013-03-11 | 2013-07-10 | 中国科学院计算机网络信息中心 | Massive geoscientific data parallel processing method based on distributed file system |
| CN103268245A (en) * | 2012-11-28 | 2013-08-28 | 北京建筑工程学院 | Immediate streamline meteorological data processing method |
| CN104268423A (en) * | 2014-10-11 | 2015-01-07 | 武汉大学 | Large-scale dynamic evolution dust type aerosol retrieval method |
-
2015
- 2015-11-06 CN CN201510747043.0A patent/CN105426235B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6909815B2 (en) * | 2003-01-31 | 2005-06-21 | Spectral Sciences, Inc. | Method for performing automated in-scene based atmospheric compensation for multi-and hyperspectral imaging sensors in the solar reflective spectral region |
| CN102778675A (en) * | 2012-04-28 | 2012-11-14 | 中国测绘科学研究院 | Atmospheric correction method and atmospheric correction module for satellite remote-sensing image |
| CN103268245A (en) * | 2012-11-28 | 2013-08-28 | 北京建筑工程学院 | Immediate streamline meteorological data processing method |
| CN103020325A (en) * | 2013-01-17 | 2013-04-03 | 中国科学院计算机网络信息中心 | Distributed remote sensing data organization query method based on NoSQL database |
| CN103198097A (en) * | 2013-03-11 | 2013-07-10 | 中国科学院计算机网络信息中心 | Massive geoscientific data parallel processing method based on distributed file system |
| CN104268423A (en) * | 2014-10-11 | 2015-01-07 | 武汉大学 | Large-scale dynamic evolution dust type aerosol retrieval method |
Non-Patent Citations (1)
| Title |
|---|
| 唐家奎: "基于MODIS数据气溶胶反演建模与网格计算中间件研究", 《中国优秀博硕士学位论文全文数据库 (博士) 工程科技I辑》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105426235B (en) | 2018-09-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240821B (en) | Distributed cross-domain collaborative computing and service system and method based on edge computing | |
| CN111581291A (en) | Data processing method and device, electronic equipment and readable medium | |
| CN103473696A (en) | Method and system for collecting, analyzing and distributing internet business information | |
| EP2959387B1 (en) | Method and system for providing high availability for state-aware applications | |
| Maciel et al. | A survey on reliability and availability modeling of edge, fog, and cloud computing | |
| CN113867600A (en) | Development method and device for processing streaming data and computer equipment | |
| CN103530168A (en) | Multi-satellite remote sensing data processing system and method based on virtualization technology | |
| Alaasam et al. | Scientific micro-workflows: where event-driven approach meets workflows to support digital twins | |
| CN111552494B (en) | Method, device, system and medium for managing container group | |
| Yang et al. | A scalable multi-data sources based recursive approximation approach for fast error recovery in big sensing data on cloud | |
| Gómez Canaval et al. | NPEPE: Massive natural computing engine for optimally solving NP-complete problems in Big Data scenarios | |
| Liu et al. | Distributed service integration for disaster monitoring sensor systems | |
| Haunert et al. | Optimal simplification of building ground plans | |
| CN105426235A (en) | Depicting method for terrestrial atmospheric aerosol retrieval distributed workflow dependency | |
| Wei et al. | OpenCluster: a flexible distributed computing framework for astronomical data processing | |
| Oparin et al. | Automated tools for the development of microservice compositions for hybrid scientific computations. | |
| CN110769037B (en) | Resource allocation method for embedded edge computing platform | |
| Ahn et al. | A Bigdata Search Engine Based Cloud Computing Network Architecture in Smart Factory Environment | |
| Kaepke et al. | A comparative evaluation of big data frameworks for graph processing | |
| MacLaren et al. | Shelter from the storm: Building a safe archive in a hostile world | |
| CN111597156A (en) | Civil aviation data processing method and device, electronic equipment and storage medium | |
| CN104572649A (en) | Data processing method, device and system for distributed storage system | |
| Prasanna | Goffish: Graph-oriented framework for foresight and insight using scalable heuristics | |
| CN114742707B (en) | Multi-source remote sensing image splicing method and device, electronic equipment and readable medium | |
| Sankar Ramachandran et al. | Distributed computing for internet of things under adversarial environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180925 Termination date: 20211106 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |