CN106022245A - Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification - Google Patents
Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification Download PDFInfo
- Publication number
- CN106022245A CN106022245A CN201610322284.5A CN201610322284A CN106022245A CN 106022245 A CN106022245 A CN 106022245A CN 201610322284 A CN201610322284 A CN 201610322284A CN 106022245 A CN106022245 A CN 106022245A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- parallel processing
- remote sensing
- data
- satellite data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Abstract
Provided is a multi-source remote sensing satellite data parallel processing system and method based on algorithm classification. The system and method support new algorithm register, store and manage the registered algorithm, select one or more algorithms from a algorithm register module, push remote sensing satellite data required to be processed and the selected algorithms to a parallel processing node to be processed; the parallel processing nodes simultaneously parallel compute the remote sensing satellite data required to be processed according to the algorithms received by nodes and an executing order; monitors the execution of the algorithms pushed to the parallel processing nodes; and files and stores processed results acquired by the parallel processing nodes. The system and the method support various remote sensing algorithm and multi-source remote sensing data, may simultaneously respond to and execute a multi-source remote sensing data processing request in a distributed cluster, solves multi-source remote sensing satellite data multi-algorithm comprehensive processing, parallel computing, and distributed storage, and achieve a multi-source remote sensing satellite data parallel processing effect.
Description
Technical field
The present invention relates to a kind of multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification and side
Method, belongs to satellite data and processes and application.
Background technology
Satellite data processes and refers to that the initial data passed down by satellite becomes corresponding image through processing,
And from satellite data, extract various information, global, high dynamic, the company having by satellite data
Continuous property, round-the-clock, round-the-clock, enriched data acquisition feature, be widely used in as agricultural, forestry, water
The professional fields such as profit, mapping, traffic, meteorology, ocean provide data support.Its object is to can be
Under limited hardware condition, stablize as far as possible, particular data Processing Algorithm is performed quickly, produce high-quality
The satellite data product of amount.
Through development for many years, China's satellite information acquisition, process and application technology achieve significantly one-tenth
Achievement, has gradually formed meteorology, resource, ocean, environment disaster reduction four big civil remote sensing satellite series.?
Under the promotion of the large-scale problems such as 863 Program, high score are special, China's remote sensing application research enter maximization,
Rapid developmental stage.Meanwhile, various satellite platforms and sensor device no matter kind, quantity is also
It is that quality is all constantly promoting.The data of China's satellite remote sensing ground station reception already more than PB level at present,
And will increase with the speed more than 10TB/ days.The data volume that satellite remote sensing date is huge, needs by sea
Amount date storage method carries out filing storage.Magnanimity not only includes jumbo data storage capacities, also wraps
Include large-scale data throughput capabilities.Along with the growth of portfolio, memory data output and storage performance rapidly
Increase, also require that method has high dynamic scalable performance, it is to avoid operation system is made by storage dilatation
Growth time interrupts.On the other hand, the satellite application for the purpose of monitoring, calamity emergency etc. is for timeliness
Property demand more and more higher, to satellite data process bring new challenge.In the face of the mass data received,
Possess efficient data-handling capacity, could meet towards agriculture feelings, the condition of a fire, the condition of a disaster, environment, meteorology
Etc. the application demand of conglomerate, and due to processing equipment and the restriction of traditional technology method, still have a large amount of
Data the most effectively processed and utilized, general 50% to 90% data are in idle or half idle
State, for satellite resource and storage resource significant wastage.Additionally, the cluster parallel processing of routine solves
Scheme to ensure that the resource utilization of the highest calculating demand needs the configuration redundant computation more than a times and deposits
Storage resource, causes the period major part resource in little calculating demand to be in no-load running state.
Summary of the invention
Present invention solves the technical problem that for: overcome prior art not enough, it is provided that a kind of based on algorithm classification
Multi-source remote sensing satellite data parallel processing system (PPS) and method, to solve in prior art, remote sensing satellite number
Satellite data kind, satellite data Processing Algorithm and the sea become increasingly complex cannot be mated according to processing system
Amount data filing needs with storing, and restriction satellite data integrated treatment is sharp with the autgmentability difference of application and resource
By the low defect of rate, it is achieved the satellite data of coupling magnanimity, complicated satellite data Processing Algorithm, it is provided that
The ability of the fast parallel calculating of satellite data high-performance.
The technical scheme that the present invention provides is: a kind of multi-source remote sensing satellite data based on algorithm classification is parallel
Processing system and method, including algorithm Registering modules, algorithm pushing module, parallel processing module, task
Monitoring module, data filing module;
Algorithm Registering modules, supports new algorithm registration, the algorithm registered is stored and managed.Calculate
Method Registering modules, supports new algorithm registration, the algorithm registered is stored and managed.Described new calculation
All parameters that method mainly performs needed for process should not comprise man-machine interaction, and algorithm execution can be held at algorithm
Determine before row;
Algorithm pushing module, according to the needs of task, choose from algorithm Registering modules one of needs or
Polyalgorithm, when there being polyalgorithm, it is possible to arranges the execution sequence of polyalgorithm, according to needing
Remote sensing satellite data to be processed arrive the migration amount of each parallel processing node, multiple parallel processings are saved
Point arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing satellite data move
Several nodes that shifting amount is minimum, if these nodes have to meet carries out data process according to selected algorithm
Resource, then will need remote sensing satellite data to be processed and the algorithm chosen to deliver to these nodes;If selecting
Make that several nodes of remote sensing satellite Data Migration amount minimum have node not have to enter according to selected algorithm
The resource that row data process, then postpone and choose having according to choosing beyond these nodes that migration amount is minimum
Fixed algorithm carries out the node of the resource requirement of data process, does not have to enter according to selected algorithm to replace
The node of the resource requirement that row data process, it would be desirable to the remote sensing satellite data of process and the algorithm chosen send
To these nodes;
The algorithm that multiple parallel processing nodes in parallel processing module receive according to each node simultaneously, presses
According to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processing nodes obtain
Multiple result of calculations also store, and multiple parallel processing nodes can be according to the needs of task, again by multiple
Result of calculation double counting uses;
Mission Monitor module, enters the implementation status of the one or more algorithms delivering to each parallel processing node
Row monitoring, when task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module energy
Enough notice tasks need the next algorithm performed to carry out data process, until task needs all of execution
Algorithm terminates.
Data filing module, the result obtaining each parallel processing node carries out filing storage, energy
Enough set up the corresponding relation of result and the metadata of result, tie with processing according to this result
The corresponding relation of the metadata of fruit, it is possible to transfer the result of needs.
Mission Monitor module can monitor the resource behaviour in service of parallel processing node and record performed
The execution time of algorithm.
Described parallel processing node is multiple, and described result splits into the original of multiple 64MB
Block, and obtain replicating blocks of files by the blocks of files duplication of the plurality of 64MB, by original block and duplication literary composition
Part block is uniformly stored in all parallel processing nodes, and replicates blocks of files and original block not with in the lump
Row processes in node.
Algorithm Registering modules, when new algorithm is registered, according to the interface specification set, utilizes XML language
Write the parameter list that algorithm needs.
A kind of multi-source remote sensing satellite data method for parallel processing, comprises the following steps that
(1) algorithm Registering modules, supports new algorithm registration, the algorithm registered is stored and managed
Reason;
(2) algorithm pushing module, according to the needs of task, selects from step (1) algorithm Registering modules
Take one or more algorithms of needs, when there being polyalgorithm, it is possible to the execution sequence of polyalgorithm is entered
Row arrangement, the remote sensing satellite data processed as required arrive the migration amount of each parallel processing node, will
Multiple parallel processing nodes arrange, and according to the quantity of the parallel processing node that task needs, select to make
Several nodes that remote sensing satellite Data Migration amount is minimum, meet according to selected algorithm if these nodes have
Carry out the resource of data process, then remote sensing satellite data to be processed and the algorithm chosen will be needed to deliver to these
Node;If making of selecting the minimum several nodes of remote sensing satellite Data Migration amount have node do not have according to
Selected algorithm carries out the resource of data process, then postpone and choose beyond these nodes that migration amount is minimum
The node with the resource requirement carrying out data process according to selected algorithm, with replace do not have according to
Selected algorithm carries out the node of the resource requirement of data process, it would be desirable to the remote sensing satellite data of process and
The algorithm chosen delivers to these nodes;
(3) calculation that the multiple parallel processing nodes in parallel processing module receive according to each node simultaneously
Method, according to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processings save
Point obtains multiple result of calculation, and multiple parallel processing nodes can be according to the needs of task, again by multiple
Result of calculation double counting uses;
(4) Mission Monitor module, the execution to the one or more algorithms delivering to each parallel processing node
Situation is monitored, when task needs to perform polyalgorithm, after an algorithm completes, and Mission Monitor
Module can notify that task needs the next algorithm performed to carry out data process, until task needs to perform
All algorithms terminate.
(5) data filing module, the result obtaining each parallel processing node carries out filing and deposits
Storage, it is possible to set up the corresponding relation of the metadata of result and result, according to this result with
The corresponding relation of the metadata of result, it is possible to transfer the result of needs.
Present invention advantage compared with prior art is:
(1) present invention processes remote sensing satellite data processing algorithm is pushed to different process nodes
Before, also include: judge aforementioned remote sensing satellite data processing algorithm desired data file storage location, and select
Select the node that Data Migration amount is minimum, algorithm is migrated to the node that aforementioned Data Migration amount is minimum;
(2) in the satellite data parallel processing module of the present invention: the executed in parallel stream to described production task
Journey is managed and monitors, and resource situation and history by monitoring calculating node perform to be recorded as locating parallel
Reason strategy provides foundation;
(3) in the satellite data parallel processing module of the present invention: the result of described production task uses distribution
Formula method is stored in all nodes;
(4) distributed storage method of the present invention, also includes: the process result pigeonholing of described production task
While storage, set up the index of metadata and data according to default rule;
(5) the algorithm Registering modules of the present invention, also includes: the algorithm interface specification of standard, for meeting
The new algorithm of interface specification, it is possible to dynamically include in and participate in described remote sensing satellite data in parallel processing flow process
The execution of Processing Algorithm.
(6) present invention provide multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification and
In method, it is possible to respond simultaneously and perform the multiple place of multi-source Remote Sensing Data data on distributed computer cluster
Adjustment method is asked, and is adaptive selected the node calculating performing operation according to algorithm and data deposit position
Machine.Compared with tradition list satellite form processing method, it is possible to make full use of multiformity and the process of satellite data
The durability of algorithm, decreases and calculates the migration of data between node.This programme can support all kinds of remote sensing
Algorithm and multi-source Remote Sensing Data data, it is possible to respond on distributed type assemblies simultaneously and perform at multi-source Remote Sensing Data data
Reason request, reduces the data throughput bottleneck during multi-source satellite data processes, improves production efficiency, and solution is many
Source remote sensing satellite data many algorithm synthesis process, parallel computation and the problem of distribution storage, reach multi-source distant
The effect of sense satellite data parallel processing.
Accompanying drawing explanation
The multi-source satellite data parallel processing system (PPS) block schematic illustration that Fig. 1 provides for the present invention;
The multi-source satellite data method for parallel processing hierarchical chart that Fig. 2 provides for the present invention;
The multi-source satellite data method for parallel processing data management hierarchical chart that Fig. 3 provides for the present invention;
The marine oil overflow monitoring that Fig. 4 provides for the present invention performs schematic diagram with simulation algorithm.
Detailed description of the invention
The basic ideas of the present invention are: the invention provides a kind of multi-source remote sensing satellite based on algorithm classification
Parallel data processing system and method, supports new algorithm registration, the algorithm registered is stored and managed
Reason;According to the needs of task, from algorithm Registering modules, choose one or more algorithm, it would be desirable to process
Remote sensing satellite data and the algorithm chosen be pushed to parallel processing node and process;Multiple parallel processings
The algorithm that node receives according to each node simultaneously, according to execution sequence, to needing remote sensing satellite to be processed
Data parallel;The implementation status delivering to each parallel processing node algorithm is monitored;To each also
The result that row process node obtains carries out filing storage.This programme supports all kinds of remote sensing algorithms and multi-source
Remotely-sensed data, it is possible to respond on distributed type assemblies simultaneously and perform multi-source Remote Sensing Data data process request, solving
Certainly multi-source remote sensing satellite data many algorithm synthesis process, parallel computation and the problem of distribution storage, reach many
The effect of source remote sensing satellite parallel data processing.
The present invention is described in detail with specific embodiment below in conjunction with the accompanying drawings.
Embodiment one:
For in prior art, remote sensing satellite data handling system cannot mate the satellite number become increasingly complex
Filing according to kind, satellite data Processing Algorithm and mass data and need with storage, restriction satellite data is combined
Conjunction processes the defect low with the autgmentability of application difference and resource utilization, and the embodiment of the present application realizes coupling sea
The satellite data of amount, complicated satellite data Processing Algorithm, it is provided that the fast parallel meter of satellite data high-performance
The ability calculated.
In conjunction with system framework schematic diagram described in accompanying drawing 1, the present embodiment is described further, the method bag
Containing following steps:
Algorithm Registering modules 101, supports new algorithm registration, the algorithm registered is stored and managed.
Described new algorithm mainly performs process should not comprise man-machine interaction, and algorithm perform needed for all parameters can
Determine before algorithm performs.According to the interface specification set, XML language is utilized to write what algorithm needed
Parameter list.
Concrete, according to the feature of remote sensing algorithm, when registration algorithm, carried by reading algorithm author
The registration interface handed over, including name of product, performs program name, the manufacturing parameter of needs and explanation, for
Each answer remote sensing algorithm generate specific production procedure.
Algorithm pushing module 102, according to the needs of task, choose needs from algorithm Registering modules one
Or polyalgorithm, when there being polyalgorithm, it is possible to the execution sequence of polyalgorithm is arranged, according to
Remote sensing satellite data to be processed are needed to arrive the migration amount of each parallel processing node, by multiple parallel processings
Node arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing satellite data
Several nodes that migration amount is minimum, if these nodes have to meet carries out data process according to selected algorithm
Resource, then remote sensing satellite data to be processed and the algorithm chosen will be needed to deliver to these nodes;If selecting
Make the minimum several nodes of remote sensing satellite Data Migration amount have node not have according to selected algorithm
Carry out the resource of data process, then postpone choose beyond these nodes that migration amount is minimum have according to
Selected algorithm carries out the node of the resource requirement of data process, does not have according to selected algorithm to replace
Carry out the node of the resource requirement of data process, it would be desirable to the remote sensing satellite data of process and the algorithm chosen
Deliver to these nodes.
Parallel processing module 103, the multiple parallel processing nodes in parallel processing module are simultaneously according to each
The algorithm that node receives is according to execution sequence, to needing remote sensing satellite data parallel to be processed, many
Individual parallel processing node obtains multiple result of calculation and stores, and multiple parallel processing nodes can be according to task
Needs, again by multiple result of calculation double countings use.
Parallel processing module hierarchical chart as described in Figure 2, describes preferred parallel processing module
Implementation method, is being obtained by lower floor's distributed file system, parallel processing cluster and flow scheduling scheme
After algorithm and cluster running status, according to the strategy set, algorithm is pushed to the highest calculating of priority and saves
Point performs, and keeps the tracking to algorithm to obtain execution result information.
Parallel processing module, including model component, MDAC, infrastructure service assembly, business clothes
Business assembly, resource component.
First, data and algorithm that platform is provided by model component are converted into data model and algorithm model,
By MDAC by these models are operated, it is achieved the access to respective resources;Meanwhile,
Infrastructure service assembly use the technology such as persistence framework, IOC container to realize distributed file system and
Row processes cluster and the management of flow scheduling scheme;Business service assembly utilizes above-mentioned resource to realize various clothes
Being engaged in the service of parallel processing business, wherein, business service assembly has service management function and realizes logarithm
According to the dynamic sensing of clustered node position, algorithm place the information execution node to Processing Algorithm according to this
Carrying out dynamic assignment, algorithm push function performs to move to algorithm the concrete operations of node, task management
Function can be inquired about the algorithm information being carrying out and obtain its execution state, and business service assembly has appoints
Business scheduling feature is responsible for starting concrete algorithm and is performed task;Then, workflow component uses business service
The flow scheduling related service that assembly provides, starts parallel processing flow process;Finally, resource component pass through
Call the operation that workflow component handling process is relevant, other modules are issued parallel processing service.
Mission Monitor module 104, the execution feelings to the one or more algorithms delivering to each parallel processing node
Condition is monitored, it is possible to the algorithm that the resource behaviour in service of monitoring parallel processing node and record performed
The execution time.When task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module
The task that can notify needs the next algorithm performed to carry out data process, until task needs the institute performed
Algorithm is had to terminate.
Data filing module 105, the result obtaining each parallel processing node carries out filing storage,
The corresponding relation of result and the metadata of result can be set up, according to this result and process
The corresponding relation of the metadata of result, it is possible to transfer the result of needs.Archiving process is by described process
Result splits into the original block of multiple 64MB, and the blocks of files of the plurality of 64MB is replicated
Blocks of files, is uniformly stored in original block and duplication blocks of files in all parallel processing nodes, and replicates
Blocks of files and original block be not in same parallel processing node.
Data management hierarchical chart as described in Figure 3.Satellite data and result thereof use distributed
Method is stored in whole cluster, coordinates data retrieval, extraction, the method for statistical analysis, can manage
The attribute of data go forward side by side row space retrieval, the satellite data being distributed in whole cluster and product data are copied
Shellfish, to the position specified, keeps the monitoring to mass data information, thus provides data to prop up for step S101
Support, and ensure that minimum data postpones.
Embodiment two:
Supporting new algorithm registration described in embodiment one algorithm Registering modules 101, the algorithm registered is being entered
Row stores and management, according to the interface specification set, utilizes XML language to write the parameter that algorithm needs
On the basis of list, present embodiments provide a kind of detailed XML language parameter definition, such as table 1 institute
Show, the parameter definition table 1 specific as follows of algorithm:
Table 1 parameter definition content
1. described in parameter definition, [ProductName] label is algorithm title;
2. [ManualParams] label contains each subalgorithm parameter value in need, including literary composition
Part type parameter [FileArg], value type parameter [ValueArg];
3. [ModelArgs] label contains subalgorithm all appointments parameter, if algorithm comprises many height
Operator, then also comprise [ModelArgs] set of tags of multiple correspondence in parameter definition;
4. the file of the All Files type parameter containing current subalgorithm in [FileArg] label is complete
Path;
5. [ValueArg] label contains all numerical value of current subalgorithm or character string type
Parameter.
In the present embodiment, the standardized XML markup language parameter list to algorithm is needed is used to carry out
Definition, is not related to concrete Parameter File in algorithm invoked procedure and resolves or parameter type judgement, especially
It is applicable to remote sensing algorithm process inter-trade, that data source is complicated, there is good versatility, and to future
More complicated remote sensing algorithm has extensibility.
Embodiment three:
At needs according to task described in embodiment one algorithm pushing module 102, from algorithm Registering modules
On the basis of choosing one or more algorithms of needs, present embodiments provide a kind of for different remote sensing calculations
The concrete Selection Strategy of method.For complicated satellite data Processing Algorithm, according to strategy set in advance,
Being decomposed into one or more serial subalgorithm having certain trigger mechanism, each subalgorithm can be described as
The primitive form of parallel computation maps (Map) and abbreviation (Reduce).Such as, marine oil overflow
Emulation and storm tide monitor scheduling algorithm, will be decomposed into a series of subalgorithm chains of certain trigger mechanism;
EOS Soil Water Content inversion algorithm can be considered as only comprising a subalgorithm.
Wherein, for different algorithm types, following decomposition can be had to set tactful:
1. single scape independent process class algorithm: same algorithm repeatedly calls as subalgorithm, concrete with algorithm
Being constrained to trigger mechanism, the different nodes being assigned in cluster perform.Algorithm perform result by
Repeatedly call result and collect filing.
2. more than scape multidate integrated treatment class algorithm: algorithm is decomposed into the circulation of multiple containment mapping abbreviation
Subalgorithm, according to nearby principle and node load, selects Data Migration less and light load
Node, specifically constrain in multiple node according to algorithm and perform all subalgorithms, finally simultaneously
As MapReduce circulation, whole algorithm is obtained algorithm execution result file.
The most semi-automatic interactive remote teaching: according to algorithm specific features, takes out and is applicable in algorithm locate parallel
The part of reason, using this part again as independent algorithm with reference to single scape independent process class algorithm or
Many scapes multidate integrated treatment class algorithm policy carries out parallel processing.Mutual part is needed to lead to
Cross virtualized mode independent operating.
Present embodiments provide the classification policy for dissimilar remote sensing Processing Algorithm, concrete according to algorithm
Execution process and call data the corresponding cluster resource of feature configuration and perform flow process, can tackle more
Polymorphic type, the algorithm of more complicated execution process, it is simple to suitably perform flow process, energy for algorithms of different distribution
Enough improve execution efficiency and the effect of polynary remote sensing satellite data processing algorithm.
Embodiment four:
In the application, the execution for multi-source remote sensing satellite data parallel processing algorithm is a dynamic call
With the process of monitoring, definition when needing continuous acquisition algorithm perform state and register according to algorithm controls to calculate
The execution process of method.The algorithm present embodiments providing marine oil overflow monitoring and simulation algorithm calls and data
Stream embodiment, as described in Figure 4.
1. oil overflowing remote sense area extraction subalgorithm, inputs oil spilling regional remote sensing data, utilizes categorised decision tree
Algorithm, has trained the decision tree classification to remotely-sensed data by classification samples, and then has extracted
Oil spill area information.Subalgorithm is suitable for parallel, directly obtains the remotely-sensed data of distributed storage,
Parallel processing on reason node throughout.
2. oil spilling analogue simulation subalgorithm, for oil spilling emulation data (comprise emulation oil spilling data and ocean current,
The data such as weather), use ECOM model, complete the drift of oil spill events elaioleucite and wind
The simulation of change process, it is thus achieved that continuous time section oil spill area information;
3.DDDAS data-driven subalgorithm, first, to the remote sensing obtained in step 1 and step 2
Extract area and do Data Integration with ECOM emulation area, it is thus achieved that more accurate oil spill area,
And drawn oil spilling initial condition by neural network algorithm, and then it is the most accurate to carry out subsequent time
Oil spilling analogue simulation, and the remote sensing oil spill area combining subsequent time show that quality evaluation is tied
Really;
4. dynamic result synthon algorithm, after abovementioned steps 2 and step 3 have performed, by time multiple
The oil spilling emulation area of phase is depicted as dynamic GIF image;
5. the oil spill area of quality evaluation subalgorithm, the oil spill area that phantom is drawn and Remotely sensed acquisition
Do Overlap Analysis, draw spilled oil simulation precision and diffusion tendency accuracy.Input and overflow for remote sensing
Oil extracts area, is output as area coincident ratio and diffusion tendency accuracy.
The said method that the present embodiment is provided is according to the execution feature of marine oil overflow monitoring with simulation algorithm
Algorithm decomposed and recombinates, improve utilization rate and the execution efficiency of algorithm calculating resource.
Embodiment five:
In the present invention, all kinds of remote sensing satellite data processing algorithms are tested respectively, are formed as follows
Conclusion:
1. adhere to separately described in support matrix 2 remote sensing, agricultural, forestry, water conservancy, mapping, traffic, meteorology,
The multi-source remote sensing satellite data Processing Algorithm in Deng Ge field, ocean.
2. improve algorithm execution efficiency, process compared to unit, two conditions processing node
Lower algorithm execution efficiency is about 200%, and under 8 node condition, algorithm execution efficiency is about
800%, it is linear with PC cluster capability improving that the present invention is capable of algorithm execution efficiency
Promote.
3. the present invention improves resource utilization, it is possible to quick collecting data resource also makes full use of institute
There is the disposal ability processing node.
Table 2
Embodiment six:
Corresponding to the system described in above-described embodiment, the present embodiment additionally provides a kind of multi-source remote sensing satellite number
According to method for parallel processing, comprise the following steps that
(1) new algorithm registration, stores the algorithm registered and manages;
(2) according to the needs of task, from step (1), one or more algorithms of needs are chosen, when
When having polyalgorithm, it is possible to the execution sequence of polyalgorithm is arranged, the remote sensing processed as required
Satellite data arrives the migration amount of each parallel processing node, is arranged by multiple parallel processing nodes,
According to the quantity of the parallel processing node that task needs, select to make the several of remote sensing satellite Data Migration amount minimum
Individual node, if these nodes have meets the resource carrying out data process according to selected algorithm, then need to
Remote sensing satellite data to be processed and the algorithm chosen deliver to these nodes;If select makes remote sensing satellite number
Data process is carried out according to selected algorithm according to several nodes that migration amount is minimum have node not have
Resource, then postpone and choose having beyond these nodes that migration amount is minimum and carry out according to selected algorithm
The node of the resource requirement that data process, carries out data process to replace not have according to selected algorithm
The node of resource requirement, it would be desirable to the remote sensing satellite data of process and the algorithm chosen deliver to these nodes;
(3) algorithm that multiple parallel processing nodes receive according to each node simultaneously, according to execution sequence,
To needing remote sensing satellite data parallel to be processed, multiple parallel processing nodes obtain multiple calculating and tie
Really, multiple parallel processing nodes can be according to the needs of task, again by multiple result of calculation double countings
Use;
(4) implementation status of the one or more algorithms delivering to each parallel processing node is monitored,
When task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module can notify to appoint
Business needs the next algorithm performed to carry out data process, until task needs all algorithms knot performed
Bundle.
(5) result obtaining each parallel processing node carries out filing storage, it is possible at foundation
The corresponding relation of the metadata of reason result and result, according to first number of this result Yu result
According to corresponding relation, it is possible to transfer the result of needs.
The present embodiment is the device embodiment of embodiment one, two, three, four correspondence, and its similar part is mutual
See, do not repeat them here.
In this specification, each embodiment uses the mode gone forward one by one to describe, and each embodiment stresses
Being the difference with other embodiments, between each embodiment, identical similar portion sees mutually.
Described above to the disclosed embodiments, makes professional and technical personnel in the field be capable of or uses
The present invention.Multiple amendment to these embodiments will be aobvious and easy for those skilled in the art
Seeing, generic principles defined herein can be in the situation without departing from the spirit or scope of the present invention
Under, realize in other embodiments.Therefore, the present invention is not intended to be limited to that shown in this article these are excellent
Select embodiment, and be to fit to the widest model consistent with principles disclosed herein and features of novelty
Enclose.
Claims (5)
1. a multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification, it is characterised in that:
Return including algorithm Registering modules, algorithm pushing module, parallel processing module, Mission Monitor module, data
Shelves module;
Algorithm Registering modules, supports new algorithm registration, the algorithm registered is stored and managed, institute
State all parameters needed for algorithm performs to determine before algorithm performs;
Algorithm pushing module, according to the needs of task, choose from algorithm Registering modules one of needs or
Polyalgorithm, when there being polyalgorithm, it is possible to arranges the execution sequence of polyalgorithm, according to needing
Remote sensing satellite data to be processed arrive the migration amount of each parallel processing node, multiple parallel processings are saved
Point arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing satellite data move
Several nodes that shifting amount is minimum, if these nodes have to meet carries out data process according to selected algorithm
Resource, then will need remote sensing satellite data to be processed and the algorithm chosen to deliver to these nodes;If selecting
Make that several nodes of remote sensing satellite Data Migration amount minimum have node not have to enter according to selected algorithm
The resource that row data process, then postpone and choose having according to choosing beyond these nodes that migration amount is minimum
Fixed algorithm carries out the node of the resource requirement of data process, does not have to enter according to selected algorithm to replace
The node of the resource requirement that row data process, it would be desirable to the remote sensing satellite data of process and the algorithm chosen send
To these nodes;
The algorithm that multiple parallel processing nodes in parallel processing module receive according to each node simultaneously, presses
According to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processing nodes obtain
Multiple result of calculations also store, and multiple parallel processing nodes can be according to the needs of task, again by multiple
Result of calculation double counting uses;
Mission Monitor module, enters the implementation status of the one or more algorithms delivering to each parallel processing node
Row monitoring, when task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module energy
Enough notice tasks need the next algorithm performed to carry out data process, until task needs all of execution
Algorithm terminates;
Data filing module, the result obtaining each parallel processing node carries out filing storage, energy
Enough set up the corresponding relation of result and the metadata of result, tie with processing according to this result
The corresponding relation of the metadata of fruit, it is possible to transfer the result of needs.
A kind of multi-source remote sensing satellite data based on algorithm classification the most according to claim 1 is parallel
Processing system, it is characterised in that: Mission Monitor module can monitor the resource of parallel processing node and use shape
The execution time of the algorithm that condition and record performed.
A kind of multi-source remote sensing satellite data based on algorithm classification the most according to claim 1 is parallel
Processing system, it is characterised in that: described parallel processing node is multiple, described result is split into
The original block of multiple 64MB, and obtain replicating blocks of files by the blocks of files duplication of the plurality of 64MB,
By original block and replicate blocks of files be uniformly stored in all parallel processing nodes, and replicate blocks of files and
Original block is not in same parallel processing node.
A kind of multi-source remote sensing satellite data based on algorithm classification the most according to claim 1 is parallel
Processing system, it is characterised in that: algorithm Registering modules is when new algorithm is registered, according to the interface rule set
Model, utilizes XML language to write the parameter list that algorithm needs.
5. a multi-source remote sensing satellite data method for parallel processing based on algorithm classification, it is characterised in that
Comprise the following steps that
(1) algorithm Registering modules can support that new algorithm is registered, the algorithm registered is carried out store and
Management;
(2) algorithm pushing module is chosen from step (1) algorithm Registering modules according to the needs of task needs
The one or more algorithms wanted, when there being polyalgorithm, it is possible to arranges the execution sequence of polyalgorithm
Row, the remote sensing satellite data processed as required arrive the migration amount of each parallel processing node, by multiple
Parallel processing node arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing
Several nodes that satellite data migration amount is minimum, are carried out according to selected algorithm if these nodes have to meet
The resource that data process, then will need remote sensing satellite data to be processed and the algorithm chosen to deliver to these joints
Point;If the several nodes making remote sensing satellite Data Migration amount minimum selected have node not have according to choosing
Fixed algorithm carries out the resource of data process, then postpone and choose beyond these nodes that migration amount is minimum
There is the node of the resource requirement carrying out data process according to selected algorithm, do not have according to choosing to replace
Fixed algorithm carries out the node of the resource requirement of data process, it would be desirable to the remote sensing satellite data of process and choosing
The algorithm taken delivers to these nodes;
(3) calculation that the multiple parallel processing nodes in parallel processing module receive according to each node simultaneously
Method, according to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processings save
Point obtains multiple result of calculation, and multiple parallel processing nodes can be according to the needs of task, again by multiple
Result of calculation double counting uses;
(4) the execution feelings of the Mission Monitor module one or more algorithms to delivering to each parallel processing node
Condition is monitored, when task needs to perform polyalgorithm, after an algorithm completes, and Mission Monitor mould
Block can notify that task needs the next algorithm performed to carry out data process, until task needs execution
All algorithms terminate;
(5) data filing module carries out filing storage to the result that each parallel processing node obtains,
The corresponding relation of result and the metadata of result can be set up, according to this result and process
The corresponding relation of the metadata of result, it is possible to transfer the result of needs.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610322284.5A CN106022245B (en) | 2016-05-16 | 2016-05-16 | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610322284.5A CN106022245B (en) | 2016-05-16 | 2016-05-16 | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106022245A true CN106022245A (en) | 2016-10-12 |
CN106022245B CN106022245B (en) | 2019-09-06 |
Family
ID=57097334
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610322284.5A Active CN106022245B (en) | 2016-05-16 | 2016-05-16 | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106022245B (en) |
Cited By (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106874067A (en) * | 2017-01-24 | 2017-06-20 | 华南理工大学 | Parallel calculating method, apparatus and system based on lightweight virtual machine |
CN107315639A (en) * | 2017-05-17 | 2017-11-03 | 中国科学院遥感与数字地球研究所 | Spectrum remote-sensing image data organization method and system during a kind of high based on cluster |
CN107609702A (en) * | 2017-09-15 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of process meteorological data method and device |
CN108345497A (en) * | 2018-01-17 | 2018-07-31 | 千寻位置网络有限公司 | GNSS positions execution method and system, the positioning device of simulation offline |
CN108985709A (en) * | 2018-06-26 | 2018-12-11 | 中国科学院遥感与数字地球研究所 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
CN109150938A (en) * | 2017-06-27 | 2019-01-04 | 哈尔滨航天恒星数据系统科技有限公司 | Satellite application public service platform based on cloud service |
CN109344188A (en) * | 2018-09-11 | 2019-02-15 | 北京航空航天大学 | Remote sensing satellite efficiency artificial intelligence statistical method |
CN109558937A (en) * | 2017-09-27 | 2019-04-02 | 三星电子株式会社 | The operating method of nerve network system and nerve network system |
CN109960573A (en) * | 2018-12-29 | 2019-07-02 | 天津南大通用数据技术股份有限公司 | A kind of cross-domain calculating task dispatching method and system based on Intellisense |
CN111680889A (en) * | 2020-05-20 | 2020-09-18 | 中国地质大学(武汉) | Offshore oil leakage source positioning method and device based on cross entropy |
CN111722635A (en) * | 2020-06-05 | 2020-09-29 | 北京空间飞行器总体设计部 | Method for parallel processing tasks of remote sensing satellite and remote sensing satellite system |
CN111726592A (en) * | 2020-06-30 | 2020-09-29 | 北京市商汤科技开发有限公司 | Method and apparatus for obtaining architecture of image signal processor |
CN112308443A (en) * | 2020-11-09 | 2021-02-02 | 中国科学院空天信息创新研究院 | Batch scheduling method and device for remote sensing information product generation workflow |
CN112368995A (en) * | 2018-06-21 | 2021-02-12 | 西门子股份公司 | System for data analysis using local device and cloud computing platform |
CN112463739A (en) * | 2019-09-09 | 2021-03-09 | 山东省计算中心(国家超级计算济南中心) | Data processing method and system based on ocean mode ROMS |
CN112612617A (en) * | 2020-12-30 | 2021-04-06 | 东方红卫星移动通信有限公司 | Satellite telemetry data processing method and system and constellation state monitoring platform |
CN112632113A (en) * | 2020-12-31 | 2021-04-09 | 北京九章云极科技有限公司 | Operator management method and operator management system |
WO2021129619A1 (en) * | 2019-12-27 | 2021-07-01 | 中兴通讯股份有限公司 | Detection method and device based on laser radar, and computer readable storage medium |
CN113641482A (en) * | 2021-08-31 | 2021-11-12 | 联通(广东)产业互联网有限公司 | AI algorithm off-line scheduling method, system, computer equipment and storage medium |
CN114461357A (en) * | 2021-12-22 | 2022-05-10 | 中国科学院空天信息创新研究院 | Remote sensing satellite raw data real-time processing flow scheduling engine |
CN114489957A (en) * | 2022-04-01 | 2022-05-13 | 国家卫星海洋应用中心 | Remote sensing satellite data processing method and device and electronic equipment |
CN114510297A (en) * | 2022-03-31 | 2022-05-17 | 国家卫星海洋应用中心 | Satellite data reprocessing method and device and electronic equipment |
CN117056088A (en) * | 2023-10-11 | 2023-11-14 | 武汉大学 | Multi-mode mapping data distributed parallel computing method and system based on MapReduce |
CN117573730A (en) * | 2024-01-16 | 2024-02-20 | 腾讯科技(深圳)有限公司 | Data processing method, apparatus, device, readable storage medium, and program product |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110928692B (en) * | 2020-01-23 | 2020-08-07 | 飞诺门阵(北京)科技有限公司 | Task processing method and device and electronic equipment |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101814029A (en) * | 2010-04-20 | 2010-08-25 | 中国科学院对地观测与数字地球科学中心 | Building method capable of expanding processing function quickly in remote sensing image processing system |
US8145677B2 (en) * | 2007-03-27 | 2012-03-27 | Faleh Jassem Al-Shameri | Automated generation of metadata for mining image and text data |
CN104299241A (en) * | 2014-10-30 | 2015-01-21 | 武汉大学 | Remote sensing image significance target detection method and system based on Hadoop |
US9152881B2 (en) * | 2012-09-13 | 2015-10-06 | Los Alamos National Security, Llc | Image fusion using sparse overcomplete feature dictionaries |
CN105094984A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Resource scheduling method and system |
-
2016
- 2016-05-16 CN CN201610322284.5A patent/CN106022245B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8145677B2 (en) * | 2007-03-27 | 2012-03-27 | Faleh Jassem Al-Shameri | Automated generation of metadata for mining image and text data |
CN101814029A (en) * | 2010-04-20 | 2010-08-25 | 中国科学院对地观测与数字地球科学中心 | Building method capable of expanding processing function quickly in remote sensing image processing system |
CN101814029B (en) * | 2010-04-20 | 2013-11-27 | 中国科学院对地观测与数字地球科学中心 | Building method capable of expanding processing function quickly in remote sensing image processing system |
US9152881B2 (en) * | 2012-09-13 | 2015-10-06 | Los Alamos National Security, Llc | Image fusion using sparse overcomplete feature dictionaries |
CN104299241A (en) * | 2014-10-30 | 2015-01-21 | 武汉大学 | Remote sensing image significance target detection method and system based on Hadoop |
CN105094984A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Resource scheduling method and system |
Non-Patent Citations (1)
Title |
---|
张树凡 等: "基于云计算的多源遥感数据服务系统研究", 《现代电子技术》 * |
Cited By (34)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106874067B (en) * | 2017-01-24 | 2020-10-02 | 华南理工大学 | Parallel computing method, device and system based on lightweight virtual machine |
CN106874067A (en) * | 2017-01-24 | 2017-06-20 | 华南理工大学 | Parallel calculating method, apparatus and system based on lightweight virtual machine |
CN107315639A (en) * | 2017-05-17 | 2017-11-03 | 中国科学院遥感与数字地球研究所 | Spectrum remote-sensing image data organization method and system during a kind of high based on cluster |
CN109150938A (en) * | 2017-06-27 | 2019-01-04 | 哈尔滨航天恒星数据系统科技有限公司 | Satellite application public service platform based on cloud service |
CN107609702A (en) * | 2017-09-15 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of process meteorological data method and device |
CN109558937A (en) * | 2017-09-27 | 2019-04-02 | 三星电子株式会社 | The operating method of nerve network system and nerve network system |
CN109558937B (en) * | 2017-09-27 | 2023-11-28 | 三星电子株式会社 | Neural network system and operation method of neural network system |
CN108345497A (en) * | 2018-01-17 | 2018-07-31 | 千寻位置网络有限公司 | GNSS positions execution method and system, the positioning device of simulation offline |
CN112368995A (en) * | 2018-06-21 | 2021-02-12 | 西门子股份公司 | System for data analysis using local device and cloud computing platform |
CN108985709A (en) * | 2018-06-26 | 2018-12-11 | 中国科学院遥感与数字地球研究所 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
CN109344188A (en) * | 2018-09-11 | 2019-02-15 | 北京航空航天大学 | Remote sensing satellite efficiency artificial intelligence statistical method |
CN109960573A (en) * | 2018-12-29 | 2019-07-02 | 天津南大通用数据技术股份有限公司 | A kind of cross-domain calculating task dispatching method and system based on Intellisense |
CN109960573B (en) * | 2018-12-29 | 2021-01-08 | 天津南大通用数据技术股份有限公司 | Cross-domain computing task scheduling method and system based on intelligent perception |
CN112463739A (en) * | 2019-09-09 | 2021-03-09 | 山东省计算中心(国家超级计算济南中心) | Data processing method and system based on ocean mode ROMS |
WO2021129619A1 (en) * | 2019-12-27 | 2021-07-01 | 中兴通讯股份有限公司 | Detection method and device based on laser radar, and computer readable storage medium |
CN111680889A (en) * | 2020-05-20 | 2020-09-18 | 中国地质大学(武汉) | Offshore oil leakage source positioning method and device based on cross entropy |
CN111680889B (en) * | 2020-05-20 | 2023-08-18 | 中国地质大学(武汉) | Cross entropy-based offshore oil leakage source positioning method and device |
CN111722635A (en) * | 2020-06-05 | 2020-09-29 | 北京空间飞行器总体设计部 | Method for parallel processing tasks of remote sensing satellite and remote sensing satellite system |
CN111726592B (en) * | 2020-06-30 | 2022-06-21 | 北京市商汤科技开发有限公司 | Method and apparatus for obtaining architecture of image signal processor |
CN111726592A (en) * | 2020-06-30 | 2020-09-29 | 北京市商汤科技开发有限公司 | Method and apparatus for obtaining architecture of image signal processor |
CN112308443A (en) * | 2020-11-09 | 2021-02-02 | 中国科学院空天信息创新研究院 | Batch scheduling method and device for remote sensing information product generation workflow |
CN112612617B (en) * | 2020-12-30 | 2023-06-20 | 东方红卫星移动通信有限公司 | Satellite telemetry data processing method and system and constellation state monitoring platform |
CN112612617A (en) * | 2020-12-30 | 2021-04-06 | 东方红卫星移动通信有限公司 | Satellite telemetry data processing method and system and constellation state monitoring platform |
CN112632113A (en) * | 2020-12-31 | 2021-04-09 | 北京九章云极科技有限公司 | Operator management method and operator management system |
CN113641482B (en) * | 2021-08-31 | 2024-03-22 | 联通(广东)产业互联网有限公司 | AI algorithm offline scheduling method, system, computer equipment and storage medium |
CN113641482A (en) * | 2021-08-31 | 2021-11-12 | 联通(广东)产业互联网有限公司 | AI algorithm off-line scheduling method, system, computer equipment and storage medium |
CN114461357A (en) * | 2021-12-22 | 2022-05-10 | 中国科学院空天信息创新研究院 | Remote sensing satellite raw data real-time processing flow scheduling engine |
CN114461357B (en) * | 2021-12-22 | 2022-11-11 | 中国科学院空天信息创新研究院 | Remote sensing satellite original data real-time processing flow scheduling system |
CN114510297A (en) * | 2022-03-31 | 2022-05-17 | 国家卫星海洋应用中心 | Satellite data reprocessing method and device and electronic equipment |
CN114489957A (en) * | 2022-04-01 | 2022-05-13 | 国家卫星海洋应用中心 | Remote sensing satellite data processing method and device and electronic equipment |
CN117056088A (en) * | 2023-10-11 | 2023-11-14 | 武汉大学 | Multi-mode mapping data distributed parallel computing method and system based on MapReduce |
CN117056088B (en) * | 2023-10-11 | 2024-01-19 | 武汉大学 | Multi-mode mapping data distributed parallel computing method and system based on MapReduce |
CN117573730A (en) * | 2024-01-16 | 2024-02-20 | 腾讯科技(深圳)有限公司 | Data processing method, apparatus, device, readable storage medium, and program product |
CN117573730B (en) * | 2024-01-16 | 2024-04-05 | 腾讯科技(深圳)有限公司 | Data processing method, apparatus, device, readable storage medium, and program product |
Also Published As
Publication number | Publication date |
---|---|
CN106022245B (en) | 2019-09-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106022245A (en) | Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification | |
CN112115198B (en) | Urban remote sensing intelligent service platform | |
CN103425772B (en) | A kind of mass data inquiry method with multidimensional information | |
CN111680025A (en) | Method and system for intelligently assimilating space-time information of multi-source heterogeneous data oriented to natural resources | |
CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
CN106372114A (en) | Big data-based online analytical processing system and method | |
CN103631922B (en) | Extensive Web information extracting method and system based on Hadoop clusters | |
CN102722355A (en) | Workflow mechanism-based concurrent ETL (Extract, Transform and Load) conversion method | |
CN103631657A (en) | Task scheduling algorithm based on MapReduce | |
CN103605662A (en) | Distributed computation frame parameter optimizing method, device and system | |
CN107247799A (en) | Data processing method, system and its modeling method of compatible a variety of big data storages | |
CN107515952A (en) | The method and its system of cloud data storage, parallel computation and real-time retrieval | |
CN109299298A (en) | Construction method, device, application method and the system of image fusion model | |
CN106202378A (en) | The immediate processing method of a kind of streaming meteorological data and system | |
CN103177035A (en) | Data query device and data query method in data base | |
CN106875320A (en) | The efficient visual analysis method of ship aeronautical data under cloud environment | |
CN106991135A (en) | Towards the quick tile generation method of remote sensing image data | |
CN107944765A (en) | Intelligence manufacture production scheduling cooperates with the assessment system and appraisal procedure of management and control ability | |
CN107463151B (en) | A kind of complex surface machining multidimensional knowledge cloud cooperating service method | |
CN112948123A (en) | Spark-based grid hydrological model distributed computing method | |
CN103345485B (en) | A kind of mainframe platform dynamic statement automatic generation method and system | |
CN110048886A (en) | A kind of efficient cloud configuration selection algorithm of big data analysis task | |
CN115170924A (en) | Intelligent interpretation system for air, space and ground big data | |
Zhou et al. | Research on the Internet of Things Platform Design for Agricultural Machinery Operation and Operation Management | |
CN107122849A (en) | Product checking total complete time minimization method based on SparkR |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |