CN105426235B - A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method - Google Patents
A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method Download PDFInfo
- Publication number
- CN105426235B CN105426235B CN201510747043.0A CN201510747043A CN105426235B CN 105426235 B CN105426235 B CN 105426235B CN 201510747043 A CN201510747043 A CN 201510747043A CN 105426235 B CN105426235 B CN 105426235B
- Authority
- CN
- China
- Prior art keywords
- aerosol
- data
- task
- node
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method, it is related to high-performance geoscience computing field, the especially MOSDIS data towards cloud computing, by building a multi-component system, member is wanted to describe a finite graph being distributed on network from mathematical form with 6, carrying out mathematical form to aerosol remote-sensing inversion distributed work flow dependence portrays；Pass through the interpretation to being abstracted depicting method, after obtaining related remote sensing processing task, configuration, control information, abstract service flow is subjected to resource impact, obtain remote-sensing inversion processing and the data processing related service of Web Service encapsulation, execution of the driving aerosol inversion procedure process under distributed computing environment.In the flow definition stage, Web Services are called；By pair with bifurcated, with merge or bifurcated or merge four kinds of constructing modules and carry out circulation control, realization is orderly executed according to parallel with serial, precursor and successor relationship in distributed computer environment, realizes the fast inversion of aerosol.
Description
Technical field
The present invention relates to high-performance geoscience computing fields, and especially the MODIS data land air gas towards cloud computing is molten
Glue inverting distributed work flow dependence depicting method.
Background technology
MODIS（The abbreviation of English moderate-resolution imaging spectroradiometer, full name are
Moderate Imaging Spectroradiomete）Data land Atmospheric Aerosol Retrieval with Remote Sensing Technique needs to carry out a large amount of data prediction and complexity
The calculating of radiation transfer equation, operand is big, and time-consuming.These processing procedures include mainly calibration, radiation transmission, set school
It is positive to wait data predictions, the image procossings such as band math, image transformation, data fusion, Solving Nonlinear Systems of Equations isoinversion fortune
The operations such as calculation and post-processing.Use the Distributed Workflow Technology towards the atmospheric aerosol inverting of MOSDIS data land can be with
These process steps are decomposed, formed has the subtask of dependence between each other, and according to subtask precursor with it is follow-up,
It is run in distributed computer environment with Serial Relation parallel, and then whole inverting processing time can be reduced, to real
The fast inversion of existing aerosol.When these processes are described, using modes such as natural language or data flow diagram, can cause
Conflicting statement and the understanding to stating different modes almost invariably will appear ambiquity, and these are described
Mode is imperfect, and abstraction hierarchy is chaotic.Therefore it needs to carry out succinctly accurately the above process using the method for mathematical form
Being seamlessly transitted between different soft project activities as a result, realizing for its state, phenomenon or action is described, but in the prior art
How the method that how not realize realizes it is the technical issues of those skilled in the art constantly explore.
Invention content
It is an object of the invention to overcome the deficiencies of the prior art and provide a kind of land atmospheric aerosol inverting distribution works
Make stream dependence depicting method, realizes in distributed computer environment according to parallel and serial, precursor and successor relationship
It orderly executes, realizes the fast inversion of aerosol.
In order to achieve the above object, the technical solution adopted by the present invention is：
A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method, the method is towards MODIS
Data include the following steps：
Step 1. according to finite graph DND=（S, T, I, O, N, D）The remote-sensing inversion service flow describer of rule structure is molten to gas
Glue quantitative inversion processing step carries out abstract description, is formed with the abstract service stream of dependence；
Step 2. services stream analyzer and is carried out according to task, configuration information, the several classifications of control information to abstract service stream
Parsing, is decomposed into the identifiable relevant parameter of Service Matching device, and abstract service stream is carried out resource impact；
Step 3. Service Matching device according to the task of acquisition, match information, control information, with UDDI service register centers,
Metadata catalog, deblocking management, task manager are communicated, scheduling correlation module operation；
Step 4. task manager is according to the address of service of acquisition, data block address, control information, configuration information, driving
Actuator sequentially, parallel/serial row execute Web service stream（That is web service stream）, and implementation procedure is monitored, it reports an error to abnormal, and
Restart execution；
When step 5. actuator encounters aerosol inversion procedure service call point in the process of implementation, in service registration library
It finds corresponding Web service and is sent to actuator, carry out example execution, and obtain handling result；
In the process of implementation, actuator takes out the service of aerosol inversion procedure to tie up to step 6. from service registration library successively
It sets the tone use；When an aerosol inversion procedure service call fails, it is deleted from service registration library, and trigger exception
Reason device is handled；
Step 7. repeats step 3-6, until flow is executed to without subsequent point of invocation, completes the inverting of atmospheric aerosol；
The DND rules want member one finite graph being distributed on network of description by six, and DND=（S, T, I, O,
N, D）；Wherein S={ s1,s2...,spDescription aerosol remote-sensing inversion Processing Algorithm state Finite set, p>0 is the number of S collection；
T={t1,t2...,tkDescription transfer process this finite aggregate, k>0 is the number of transfer process；N={n1,n2...,nqIt is net
Network back end IP address collection, q>0 is back end number in network；D={d1,d2...,duIt is data set and its segmentation collection, u
>0 is a MODIS HDF（HDF is the abbreviation of Hierarchical Data Format, can store different types of image
With the file format of digital data）Formatted file and its data block number of segmentation；I:D×S×T->Z is input function, is defined
The HDFS being distributed on different data node（That is Hadoop distributed file systems）Directed arc of the data file in S to T
The set of repeat number or power, Z={ 0,1 ... } is set of nonnegative integer herein；O:T×D×S->N is output function, defines gas
Colloidal sol remote-sensing inversion Processing Algorithm is converted on different data node, and handling result is remapped to the mistake of heterogeneous networks
The set of journey.
As presently preferred embodiments of the present invention, S of the present invention={ s1,s2...,spIn the quantitative distribution of aerosol remote sensing
During service flow operating status is portrayed, including following processing step：
Step S101. pair warp and wefts degree, solar zenith angle, sensor zenith angle wave band carry out MODIS data extractions；
Step S102., according to 10KM, 1KM, 500M, carries out longitude and latitude and zenith to original MODIS 1KM resolution datas
Angular data resampling；
Step S103. carries out MODIS image cuttings according to survey region, according to longitude and latitude to zenith angle；
Step S104. carries out MODIS image joints according to solar zenith angle, sensor zenith angle；
Step S105. carries out geometric correction according to longitude and latitude to solar zenith angle, sensor zenith angle；
Step S106. extracts cloud mask information from MODIS original documents；
Performance reflectivity of the step S107. from original MODIS extracting datas Band1,2,3,4,7；
Step S108. is according to ground observation point longitude and latitude and satellite transit time, the ground station of extraction and satellite synchronization
Observe data；
Step S109. calculates AOT（The abbreviation of English Advanced Oxidation Technologies, the Chinese meaning are
High-level oxidation technology）The initial value of iterative inversion；
Step S110. calculates AOT；
Compared with step S111. is surveyed with ground, inversion result is verified.
As presently preferred embodiments of the present invention, D of the present invention={ d1,d2...,duInclude that following data and segmentation collect：
Step S201. produces level one data product MOD02, MOD03 data of Moderate Imaging Spectroradiomete, air standard
Product MOD35, MOD04 data, AERNET（AErosol RObotic NETwork）These pending data sources of data are according to the time
Unit day is incorporated to same catalogue, and then according to higher level data directory is formed in the moon, year, input data set is formed with this；
S202. pairs of step with merge or merge after the intermediate treatment of hdr and img formats that generates of remote sensing special-purpose software
As a result filing processing is carried out, and task management node collection obtains archival one from T, forms intermediate filing data set；
S203. pairs of step and intermediate treatment input data hdr, img after bifurcated or bifurcated carry out deblocking, and from T
It concentrates task management node collection to obtain the idle node that N is concentrated, maps the data into these idle nodes, form deblocking
Collection；
Step S203. builds metadata management collection to centre filing data set and deblocking collection；
Step S204. whole flow process handling results form output handling result collection.
As presently preferred embodiments of the present invention, S collection of the present invention includes that ground station observes data processing, information carries
It takes, resampling, cutting, geometric correction, splicing, cloud information extraction, wave band extraction, cloud mask, AOT inverting initial values calculate, AOT is counted
It calculates and verification step is compared in ground actual measurement, the step passes through IDL（Interactive data language Interactive Data
Language）, Matlab, C++ developed, then compiling be encapsulated into web services form.
As presently preferred embodiments of the present invention, T collection of the present invention includes based on dot-product operation, field operation, global fortune
The aerosol remote-sensing inversion of calculation handles task-set, the distributed access of remote sensing source data, deblocking method, data compression side
Method, data index method, aerosol remotely-sensed data metadata management, registration, service discovery, aerosol remote-sensing inversion task pipe
Reason, mapping, aerosol remote sensing data transmission engine, node state monitoring, node tasks monitoring, node tasks configuration, node are appointed
Business is restarted, aerosol remote-sensing inversion handling result polymerize.
As presently preferred embodiments of the present invention, D of the present invention integrates as MOSDIS HDF data sets and its segmentation collection, including
Aerosol remote sensing source data, MOSDIS HDF formatted files and its partitioned data set under distributed environment, among aerosol remote sensing
Handling result collection, aerosol remote sensing handling result collection, aerosol remote sensing metadata set.
As presently preferred embodiments of the present invention, N of the present invention integrates as distributed network node management collection and back end
Collection, including Resource Management node collection, task scheduling set of node, Mission Monitor set of node, task execution set of node, back end
Collection.
As presently preferred embodiments of the present invention, I of the present invention integrates as user program according to task depicting language, execution foot
This collection, job management command set submit a task, the aerosol remote sensing that the information of task is sent in T transfer process set anti-
Drill task management node collection；Aerosol remote-sensing inversion task management node collection is the core of entire aerosol remote-sensing inversion duty mapping
The heart, need D concentrate data set with and its segmentation collection machine timed communication, manage S in aerosol remote sensing program should distinguish
It runs on the corresponding machines of which of N, needs the operations such as to manage mission failure all in N, restart；T interior joint Mission Monitors
It is the part that there is task execution node in entire distributed system, T interior joints Mission Monitor and node with node condition monitoring
The thing that condition monitoring is done is the executive condition of monitoring task and the resource situation of machine where oneself；T interior joint Mission Monitors
The operating status obtained can be sent to the aerosol remote-sensing inversion task management node collection in T by heartbeat；Aerosol remote sensing
The rally of inverting task management node collects information above and corresponds to machine to operate in which of N to the task distribution newly submitted
On；HDFS data files are in the repeat number of the directed arc of S to T or the set of power in D set, and controlling between service flow has dependence
Between the subtask of relationship precursor with it is follow-up, run execution sequence in distributed computer environment with Serial Relation parallel.
As presently preferred embodiments of the present invention, O collection of the present invention is that the processing of aerosol remote-sensing inversion is calculated in management S set
Aerosol remote-sensing inversion task management of the method in different calculating and mapping on back end and the management process really executed, T
Task ranking is carried out to each of user's submission application, passes through the calculating obtained from aerosol remotely-sensed data metadata management and provides
Source is left unused information, and the task in scheduler task sequence is further distributed to internal task, and carries out node tasks configuration, is started
Node tasks monitor, and are that task application resource progress node tasks are restarted again when task run fails；Work as node tasks
After execution, according to the HDFS data files obtained from aerosol remotely-sensed data metadata management/deblocking index mark
Will information, after the data of oneself have been calculated in the sub-line journey in N in back end, the aerosol remote-sensing inversion handling result in T is poly-
It closes main thread sub-line journey result of calculation remapped and polymerize based on the data of N data section point sets.
As presently preferred embodiments of the present invention, abstract service stream, service flow parsing described in steps 1 and 2 of the present invention, resource
Mapping process is as follows：
Step S301. carries out validation verification according to the provided relevant parameter information of service flow to abstract service stream；
Step S302. matches the task in abstract stream according to model library；
Step S303. generates running environment configuration information to the task after pairing according to the provided parameter of service flow；
Step S304. parses the task dependence after pairing；
Physical resource mapping is carried out after step S305. service flow successfully resolveds.
Compared with prior art, the beneficial effects of the invention are as follows：
The first, in specific practice of software, the process flow in distributed work flow uses Web Service skills
Art is packaged, the details of these Web Services, such as type, address are included in flow definition, passes through its structure
The general standard interface built shields each species diversity of heterogeneous platform in network；
The second, the workflow processing task of aerosol remote-sensing inversion under distributed network environment according to multi-component system DND numbers
It learns formalized description mode to be mapped, in the flow definition stage, the Web Services of calling can be each execution node
The Web Services individually developed can also be the function interface that each engine Web Services are externally provided；
Third, mathematical form description form the subtask for having dependence between each other with flow definition, and subtask exists
Pass through service call agreement SOAP, Service Description Protocol WSDL and service discovery/integrated UDDI, and clothes between distributed network
The upper layer applications such as business discovery, integrated；
4th, by pair with bifurcated, with merge bifurcated or merge four kinds of constructing modules carry out circulation control, it can be achieved that
It is orderly executed with serial, precursor and successor relationship according to parallel in distributed computer environment, realizes the quick of aerosol
Inverting.
Description of the drawings
Fig. 1 is that the aerosol remote sensing of the present invention quantifies the system architecture method flow diagram of Distributed Services stream operating status；
Fig. 2 is abstract service stream of the present invention parsing and mapping process figure.
Specific implementation mode
Idea of the invention is that overcoming the deficiencies of the prior art and provide a kind of molten towards MOSDIS data land air gas
The mathematical form depicting method of glue remote-sensing inversion distributed work flow dependence describes the dependence pass during inversion procedure
System, parallel and Serial Relation.By this mathematical form depicting method, it is anti-that related scheduling algorithm driving aerosol can be write
Drill orderly execution of the processing procedure under distributed computing environment.It is described in detail with reference to attached drawing with reference to embodiment,
So that technical characteristic and advantage of the invention are interpretated more in-depth.
Flow chart of the method for the present invention is as shown in Figure 1, the present invention establishes software architecture on the basis of mathematical form
Method is a stringent and effective approach for developing computer control system. DND=（S, T, I, O, N, D）Multi-component system uses
The mode of formalization describes between the configuration definition and these elements of component, connector and data in software architecture
Restriction relation and its attribute, it describes the migration of the behavior and state in system each stage by the way of mathematical form.DND
The system architecture that multi-component system can quantify aerosol remote sensing shown in FIG. 1 Distributed Services stream operating status carries out formalization quarter
It draws.Description process shown in Fig. 1 is as follows：
The first step, according to the remote-sensing inversion service flow describer of DND rules structure to aerosol quantitative inversion processing step
Abstract description is carried out, the abstract service stream of dependence is formed with；
Second step, service stream analyzer carry out abstract service stream according to task, configuration information, the several classifications of control information
Parsing, is decomposed into the identifiable relevant parameter of Service Matching device；
Third walk, Service Matching device according to the task of acquisition, match information, control information, with UDDI service register centers,
Metadata catalog, deblocking management, task manager are communicated, scheduling correlation module operation；
4th step, task manager is according to the address of service of acquisition, data block address, control information, configuration information, driving
Actuator sequentially, parallel/serial row execute Web service stream, and implementation procedure monitored, report an error to abnormal, and restart execution；
5th step, when actuator encounters aerosol inversion procedure service call point in the process of implementation, in service registration library
In find corresponding web services and be sent to actuator, carry out example execution, and obtain handling result；
6th step, in the process of implementation, actuator takes out the service of aerosol inversion procedure from service registration library successively to be come
Binding is called；When an aerosol inversion procedure service call fails, it is deleted from service registration library, and trigger exception
Processor is handled；
7th step, repeat third and fourth, five, six steps, until flow goes to no subsequent point of invocation.
The present invention from mathematical form, with 6 want member description one finite graph being distributed on network, DND=（S, T, I,
O, N, D）.Wherein S={ s1,s2...,spDescription aerosol remote-sensing inversion Processing Algorithm state Finite set, p>0, it is S collection
Number；T={t1,t2...,tkDescription transfer process this finite aggregate, k>0 is the number of transfer process；N={n1,n2...,nq}
For network data node IP address collection, q>0 is back end number in network；D={d1,d2...,duIt is data set and its segmentation
Collection, u>0, it is the data block number of a MOSDIS HDF formatted file and its segmentation；I:D×S×T->Z is input function, it is fixed
Justice is distributed on the HDFS data files on different data node in the repeat number of the directed arc of S to T or the set of power, here
Z=0,1 ... } it is set of nonnegative integer；O:T×D×S->N is output function, it defines aerosol remote-sensing inversion Processing Algorithm
It is converted on different data node, and handling result is remapped to the set of the process of heterogeneous networks.
Preferably, S of the present invention={ s1,s2...,spQuantify Distributed Services stream operating status in aerosol remote sensing and portray
In, mainly carried including ground station observation data processing, information extraction, resampling, cutting, geometric correction, splicing, cloud information
Take, wave band extraction, cloud mask, AOT inverting initial values calculate, AOT calculate and ground actual measurement compare verification and etc..These processing steps
Suddenly it is mainly developed using IDL, Matlab, C++, then compiling is encapsulated into web services form.
Preferably, T of the present invention={ t1,t2...,tkIn transfer process set description, it is main include based on dot-product operation,
Field operation, global operations aerosol remote-sensing inversion handle task-set, the distributed access of remote sensing source data, deblocking side
Method, data compression method, data index method, aerosol remotely-sensed data metadata management, registration, service discovery, aerosol are distant
Feel inverting task management, mapping, aerosol remote sensing data transmission engine, node state monitoring, node tasks monitoring, node tasks
Configuration, node tasks are restarted, aerosol remote-sensing inversion handling result polymerize etc..
Preferably, D of the present invention={ d1,d2...,duIt is mainly MOSDIS HDF data sets and its segmentation collection, mainly divide
Aerosol remote sensing source data, MOSDIS HDF formatted files and its partitioned data set, aerosol remote sensing middle under cloth environment
Manage result set, aerosol remote sensing handling result collection, aerosol remote sensing metadata set.
Preferably, N of the present invention={ n1,n2...,nqIt is distributed network node management collection and data set of node, including money
Source control set of node, task scheduling set of node, Mission Monitor set of node, task execution set of node, data section point set.
Preferably, I of the present invention: D×S×T->Z is user program according to task depicting language, perform script collection, operation
Administration order collection etc. submits a task, the information of task that can be sent to the aerosol remote-sensing inversion in T transfer process set and appoint
Business management node collection, aerosol remote-sensing inversion task management node collection is the core of entire aerosol remote-sensing inversion duty mapping,
It need D concentrate data set with and its segmentation collection machine timed communication, manage S in aerosol remote sensing program should run respectively
On which of N machines, need the operations such as to manage mission failure all in N, restart.T interior joints Mission Monitor and node
Condition monitoring is the part that there is task execution node in entire distributed system, and the thing that it does is mainly holding for monitoring task
The resource situation of market condition and machine where oneself.The operating status that T interior joint Mission Monitors are obtained can be sent by heartbeat
To the aerosol remote-sensing inversion task management node collection in T.These letters are collected in the rally of aerosol remote-sensing inversion task management node
Breath to the task distribution newly submitted to operate on which of N machines.Directed arc of the HDFS data files in S to T in D set
Repeat number or power set, control between the subtask for having dependence between service flow precursor with it is follow-up, parallel with string
Row relationship is run in distributed computer environment executes sequence.
Preferably, O of the present invention:T×D×S->N be management S set in aerosol remote-sensing inversion Processing Algorithm in different meters
It calculates and user is submitted with mapping on back end and the management process that really executes, the aerosol remote-sensing inversion task management in T
Each of application carry out task ranking, pass through the idle letter of the computing resource that is obtained from aerosol remotely-sensed data metadata management
It ceases, the task in scheduler task sequence is further distributed to internal task, and carries out node tasks configuration, starter node task
Monitoring, and be that task application resource progress node tasks are restarted again when task run fails.When node tasks execution terminates
Afterwards, according to the HDFS data files obtained from aerosol remotely-sensed data metadata management/deblocking index marker information, N
After the data of oneself have been calculated in sub-line journey in middle back end, the aerosol remote-sensing inversion handling result in T polymerize main thread
Sub-line journey result of calculation remapped and polymerize based on the data of N data section point sets.
In addition, S of the present invention={ s1,s2...,spQuantified during Distributed Services stream operating status portrays in aerosol remote sensing,
It include mainly following processing step：
Step S101, the wave bands such as pair warp and weft degree, solar zenith angle, sensor zenith angle carry out MODIS data extractions；
Step S102, longitude and latitude and zenith are carried out according to 10KM, 1KM, 500M to original MODIS 1KM resolution datas
Angular data resampling；
Step S103, according to survey region, MODIS image cuttings are carried out to zenith angle according to longitude and latitude；
Step S104, MODIS image joints are carried out according to solar zenith angle, sensor zenith angle；
Step S105, according to longitude and latitude, geometric correction is carried out to solar zenith angle, sensor zenith angle；
Step S106, cloud mask information is extracted from MODIS original documents；
Step S107, from the performance reflectivity of original MODIS extracting datas Band1,2,3,4,7；
Step S108, according to ground observation point longitude and latitude and satellite transit time, the ground station of extraction and satellite synchronization
Observe data；
Step S109, the initial value of AOT iterative inversions is calculated；
Step S110, AOT is calculated；
Step S111, compared with being surveyed with ground, inversion result is verified.
In order to reach preferable efficiency of inverse process, D of the present invention={ d1,d2...,duMainly there are following data and segmentation
Collection:
Step S201, by the pending data source such as MOD02, MOD03, MOD35, MOD04, AERNET according to chronomere day
It is incorporated to same catalogue, then according to higher level data directory is formed in the moon, year, input data set is formed with this；
Step S202, pair filing processing is carried out with intermediate processing results hdr, img after merging or merge, and appointed from T
Business management node collection obtains archival one, forms intermediate filing data set；
Step S203, pair deblocking is carried out with intermediate treatment input data hdr, img after bifurcated or bifurcated, and from T
Middle task management node collection obtains the idle node in N, maps the data into these idle nodes, forms deblocking collection；
Step S204, metadata management collection is built to centre filing data set and deblocking collection；
Step S205, whole flow process handling result forms output handling result collection.
The abstract service stream parsing of the present invention is with mapping process figure as shown in Fig. 2, describing abstract service stream, service flow solution
Analysis, resource impact process, it is specific as follows：
Step S301, validation verification is carried out to abstract service stream according to the provided relevant parameter information of service flow；
Step S302, the task in abstract stream is matched according to model library；
Step S303, running environment configuration information is generated according to service flow provided parameter to the task after pairing；
Step S304, the task dependence after pairing is parsed；
Step S305, physical resource mapping is carried out after service flow successfully resolved.
Compared with prior art, the present invention wants member to describe one from mathematical form by building a multi-component system with 6
The finite graph being distributed on network carries out mathematical form quarter with this to aerosol remote-sensing inversion distributed work flow dependence
It draws, including dependence during description inversion procedure, parallel and Serial Relation.Pass through the solution to this abstract depicting method
It translates, after obtaining related remote sensing processing task, configuration, control information, abstract service flow is subjected to resource impact, obtains Web
Remote-sensing inversion processing after Service encapsulation and data processing related service, driving aerosol inversion procedure process is in distribution
Computing environment under execution.In the flow definition stage, the Web Services of calling can be that each execution node is individually opened
The Web Services of hair can also be the function interface that each engine Web Services are externally provided.By pair with
Bifurcated, with merge bifurcated or merge four kinds of constructing modules carry out circulation control, it can be achieved that in distributed computer environment
In orderly executed with serial, precursor and successor relationship according to parallel, realize the fast inversion of aerosol.
Clear, complete description is carried out to the present invention by the technical solution in above example, it is clear that described reality
The embodiment that example is a present invention part is applied, instead of all the embodiments.Based on the embodiments of the present invention, this field is common
The every other embodiment that technical staff is obtained without making creative work belongs to the model that the present invention protects
It encloses.
Claims (10)
1. a kind of land atmospheric aerosol inverting distributed work flow dependence depicting method, the method is towards MODIS numbers
According to including the following steps：
Step 1. according to finite graph DND=（S, T, I, O, N, D）Remote-sensing inversion service flow constructed by formalized description method is retouched
It states device and abstract description is carried out to aerosol quantitative inversion processing step, be formed with the abstract service stream of dependence；
Step 2. services stream analyzer and is parsed according to task, configuration information, the control several classifications of information to abstract service stream,
It is decomposed into the identifiable relevant parameter of Service Matching device, abstract service stream is subjected to resource impact；
Step 3. Service Matching device is according to the task of acquisition, match information, control information, with UDDI service register centers, first number
It is communicated according to catalogue, deblocking management, task manager, scheduling correlation module operation；
Step 4. task manager is executed according to the address of service of acquisition, data block address, control information, configuration information, driving
Device sequentially, parallel/serial row execute Web service stream, and implementation procedure monitored, report an error to abnormal, and restart execution；
When step 5. actuator encounters aerosol inversion procedure service call point in the process of implementation, searched in service registration library
It is sent to actuator to corresponding Web service, carries out example execution, and obtain handling result；
In the process of implementation, actuator takes out the service of aerosol inversion procedure successively from service registration library and is adjusted to bind step 6.
With；When an aerosol inversion procedure service call fails, it is deleted from service registration library, and trigger exception handler
It is handled；
Step 7. repeats step 3-6, until flow is executed to without subsequent point of invocation, completes the inverting of atmospheric aerosol；
The finite graph DND=（S, T, I, O, N, D）Formalized description method refers to wanting one distribution of member description by six
The mathematical form of finite graph on network describes method；Wherein S={ s1,s2...,spDescription aerosol remote-sensing inversion processing
Algorithm state finite aggregate, p concentrate the number of element, and p for S>0, s1,s2...,spRespectively form aerosol remote-sensing inversion
The state of each aerosol remote-sensing inversion Processing Algorithm of Processing Algorithm state Finite set S；T={t1,t2...,tkDescription turn
Change process this finite aggregate, k>0 is the number of transfer process；N={n1,n2...,nqIt is network data node IP address collection, q
>0 is back end number in network；D={d1,d2...,duIt is data set and its segmentation collection, u>0, and u is subset in set D
Number；I:D×S×T->Z is input function, defines the HDFS data files being distributed on different data node and is arrived in S
The repeat number of the directed arc of T or the set of power, Z={ 0,1 ... } is set of nonnegative integer herein；O:T×D×S->N is output letter
Number, defines aerosol remote-sensing inversion Processing Algorithm and is converted on different data node, and handling result is remapped to
The set of the process of heterogeneous networks.
2. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 1, special
Sign is, the S={ s1,s2...,spQuantified during Distributed Services stream operating status portrays in aerosol remote sensing, including it is as follows
Processing step：
Step S101. pair warp and wefts degree, solar zenith angle, sensor zenith angle wave band carry out MODIS data extractions；
Step S102., according to 10KM, 1KM, 500M, carries out longitude and latitude and zenith angle number to original MODIS 1KM resolution datas
According to resampling；
Step S103. carries out MODIS image cuttings according to survey region, according to longitude and latitude to zenith angle；
Step S104. carries out MODIS image joints according to solar zenith angle, sensor zenith angle；
Step S105. carries out geometric correction according to longitude and latitude to solar zenith angle, sensor zenith angle；
Step S106. extracts cloud mask information from MODIS original documents；
Performance reflectivity of the step S107. from original MODIS extracting datas Band1,2,3,4,7；
Step S108. is observed according to ground observation point longitude and latitude and satellite transit time, extraction and the ground station of satellite synchronization
Data；
Step S109. calculates the initial value of AOT iterative inversions；
Step S110. calculates AOT；
Compared with step S111. is surveyed with ground, inversion result is verified.
3. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 2, special
Sign is, the D={ d1,d2...,duInclude that following data and segmentation collect：
The pending data source such as MOD02, MOD03, MOD35, MOD04, AERNET is incorporated to by step S201. according to chronomere day
Same catalogue forms input data set then according to higher level data directory is formed in the moon, year with this；
S202. pairs of step carries out filing processing, and the task pipe from T with intermediate processing results hdr, img after merging or merge
It manages set of node and obtains archival one, form intermediate filing data set；
S203. pairs of step carries out deblocking with intermediate treatment input data hdr, img after bifurcated or bifurcated, and is concentrated from T
Task management node collection obtains the idle node that N is concentrated, and maps the data into these idle nodes, forms deblocking collection；
Step S203. builds metadata management collection to centre filing data set and deblocking collection；
Step S204. whole flow process handling results form output handling result collection.
4. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 3, special
Sign is：The S collection includes ground station observation data processing, information extraction, resampling, cutting, geometric correction, splicing, cloud
Information extraction, wave band extraction, cloud mask, AOT inverting initial values calculate, AOT is calculated and verification step, the step are compared in ground actual measurement
Suddenly it is developed by IDL, Matlab, C++, then compiling is encapsulated into web services form.
5. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 4, special
Sign is：The T collection includes the aerosol remote-sensing inversion processing task-set based on dot-product operation, field operation, global operations, distant
Feel distributed access, deblocking method, data compression method, the data index method of source data, aerosol remotely-sensed data member
Data management, registration, service discovery, the task management of aerosol remote-sensing inversion, mapping, aerosol remote sensing data transmission engine, section
Dotted state monitoring, node tasks monitoring, node tasks configuration, node tasks are restarted, aerosol remote-sensing inversion handling result polymerize.
6. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 5, special
Sign is：The D integrates as data set and its segmentation collection, including aerosol remote sensing source data, MOSDIS HDF under distributed environment
Formatted file and its partitioned data set, aerosol remote sensing intermediate processing results collection, aerosol remote sensing handling result collection, aerosol are distant
Feel metadata set.
7. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 6, special
Sign is：The N integrates as distributed network node management collection and data set of node, including Resource Management node collection, task scheduling
Set of node, Mission Monitor set of node, task execution set of node, data section point set.
8. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 7, special
Sign is：The I integrates submits one to appoint as user program according to task depicting language, perform script collection, job management command set
Business, the information of task are sent to the aerosol remote-sensing inversion task management node collection in T transfer process set；Aerosol remote sensing is anti-
The core that task management node collection is entire aerosol remote-sensing inversion duty mapping is drilled, the data set of D concentrations is needed and and its is divided
Cut set machine timed communication, managing the aerosol remote sensing program in S should run respectively on the corresponding machines of which of N, need
All mission failure in management N such as restarts at the operations；T interior joints Mission Monitor is entire distributed system with node condition monitoring
The part that there is task execution node in system, the thing that T interior joints Mission Monitor is done with node condition monitoring is monitoring task
The resource situation of executive condition and machine where oneself；The operating status that T interior joint Mission Monitors are obtained can be sent out by heartbeat
Give the aerosol remote-sensing inversion task management node collection in T；More than the rally of aerosol remote-sensing inversion task management node is collected
Information to the task distribution newly submitted to operate on the corresponding machines of which of N；HDFS data files are S to T's in D set
The repeat number of directed arc or the set of power, control between the subtask for having dependence between service flow precursor with it is follow-up, simultaneously
Row runs in distributed computer environment with Serial Relation and executes sequence.
9. land atmospheric aerosol inverting distributed work flow dependence depicting method according to claim 8, special
Sign is：The O collection be management S set in aerosol remote-sensing inversion Processing Algorithm it is different calculating with back end on mapping with
The management process really executed, the aerosol remote-sensing inversion task management in T submit user each using carry out task row
Sequence, by the idle information of the computing resource obtained from aerosol remotely-sensed data metadata management, appointing in scheduler task sequence
Business is further distributed to internal task, and carries out node tasks configuration, starter node Mission Monitor, and fails in task run
Shi Chongxin is that task application resource progress node tasks are restarted；After node tasks execute, according to from aerosol remote sensing number
According to the HDFS data files obtained in metadata management/deblocking index marker information, the sub-line journey meter in N in back end
After the data for having calculated oneself, the aerosol remote-sensing inversion handling result polymerization main thread in T carries out base to sub-line journey result of calculation
It remaps and polymerize in the data of N data section point sets.
10. the land atmospheric aerosol inverting distributed work flow dependence according to any one of claim 1-9 is carved
Drawing method, it is characterised in that：Abstract service stream, service flow parsing, resource impact process specific steps described in steps 1 and 2 are such as
Under：
Step S301. carries out validation verification according to the provided relevant parameter information of service flow to abstract service stream；
Step S302. matches the task in abstract stream according to model library；
Step S303. generates running environment configuration information to the task after pairing according to the provided parameter of service flow；
Step S304. parses the task dependence after pairing；
Physical resource mapping is carried out after step S305. service flow successfully resolveds.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510747043.0A CN105426235B (en) | 2015-11-06 | 2015-11-06 | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510747043.0A CN105426235B (en) | 2015-11-06 | 2015-11-06 | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105426235A CN105426235A (en) | 2016-03-23 |
| CN105426235B true CN105426235B (en) | 2018-09-25 |
Family
ID=55504461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510747043.0A Expired - Fee Related CN105426235B (en) | 2015-11-06 | 2015-11-06 | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105426235B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6909815B2 (en) * | 2003-01-31 | 2005-06-21 | Spectral Sciences, Inc. | Method for performing automated in-scene based atmospheric compensation for multi-and hyperspectral imaging sensors in the solar reflective spectral region |
| CN102778675A (en) * | 2012-04-28 | 2012-11-14 | 中国测绘科学研究院 | Atmospheric correction method and atmospheric correction module for satellite remote-sensing image |
| CN103020325A (en) * | 2013-01-17 | 2013-04-03 | 中国科学院计算机网络信息中心 | Distributed remote sensing data organization query method based on NoSQL database |
| CN103198097A (en) * | 2013-03-11 | 2013-07-10 | 中国科学院计算机网络信息中心 | Massive geoscientific data parallel processing method based on distributed file system |
| CN103268245A (en) * | 2012-11-28 | 2013-08-28 | 北京建筑工程学院 | Immediate streamline meteorological data processing method |
| CN104268423A (en) * | 2014-10-11 | 2015-01-07 | 武汉大学 | Large-scale dynamic evolution dust type aerosol retrieval method |
-
2015
- 2015-11-06 CN CN201510747043.0A patent/CN105426235B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6909815B2 (en) * | 2003-01-31 | 2005-06-21 | Spectral Sciences, Inc. | Method for performing automated in-scene based atmospheric compensation for multi-and hyperspectral imaging sensors in the solar reflective spectral region |
| CN102778675A (en) * | 2012-04-28 | 2012-11-14 | 中国测绘科学研究院 | Atmospheric correction method and atmospheric correction module for satellite remote-sensing image |
| CN103268245A (en) * | 2012-11-28 | 2013-08-28 | 北京建筑工程学院 | Immediate streamline meteorological data processing method |
| CN103020325A (en) * | 2013-01-17 | 2013-04-03 | 中国科学院计算机网络信息中心 | Distributed remote sensing data organization query method based on NoSQL database |
| CN103198097A (en) * | 2013-03-11 | 2013-07-10 | 中国科学院计算机网络信息中心 | Massive geoscientific data parallel processing method based on distributed file system |
| CN104268423A (en) * | 2014-10-11 | 2015-01-07 | 武汉大学 | Large-scale dynamic evolution dust type aerosol retrieval method |
Non-Patent Citations (1)
| Title |
|---|
| 基于MODIS数据气溶胶反演建模与网格计算中间件研究;唐家奎;《中国优秀博硕士学位论文全文数据库 (博士) 工程科技I辑》;20051115(第07期);第1-125页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105426235A (en) | 2016-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9558033B2 (en) | Constructing and deploying patterns of flows | |
| US20150178052A1 (en) | Automated experimentation platform | |
| US20070245298A1 (en) | Apparatus and data structure for automatic workflow composition | |
| CN103733198A (en) | Stream application performance monitoring metrics | |
| US20140298472A1 (en) | Method and Apparatus for Automated Vulnerability Detection | |
| US11194686B2 (en) | Data agnostic monitoring service | |
| Saman | Monitoring and analysis of microservices performance | |
| Balbo et al. | Understanding parallel program behavior through petri net models | |
| US20180025162A1 (en) | Application program analysis apparatus and method | |
| Altintas et al. | Workflow-driven distributed machine learning in CHASE-CI: A cognitive hardware and software ecosystem community infrastructure | |
| CN106445645A (en) | Method and device for executing distributed computation tasks | |
| Krämer | A microservice architecture for the processing of large geospatial data in the cloud | |
| Olaya et al. | Building trust in earth science findings through data traceability and results explainability | |
| US11184251B2 (en) | Data center cartography bootstrapping from process table data | |
| CN105426235B (en) | A kind of land atmospheric aerosol inverting distributed work flow dependence depicting method | |
| Belgacem et al. | Virtual ez grid: A volunteer computing infrastructure for scientific medical applications | |
| Chen et al. | Apache submarine: A unified machine learning platform made simple | |
| CA2738422C (en) | Examining concurrent system states | |
| US20180095731A1 (en) | Methods and systems to identify log write instructions of a source code as sources of event messages | |
| CN108153678A (en) | A kind of test assignment processing method and processing device | |
| CN114237686A (en) | Installation package generation method and device, electronic equipment and storage medium | |
| Abdelgawad et al. | Workflow Resilience for Mission Critical Systems | |
| Li et al. | An XML based monadic framework for REST service compositions | |
| Klenik et al. | Adding semantics to measurements: Ontology-guided, systematic performance analysis | |
| Qiu et al. | Map-Reduce for Multiprocessing Large Data and Multi-threading for Data Scraping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180925 Termination date: 20211106 |