CN106022245A - Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification - Google Patents

Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification Download PDF

Info

Publication number
CN106022245A
CN106022245A CN201610322284.5A CN201610322284A CN106022245A CN 106022245 A CN106022245 A CN 106022245A CN 201610322284 A CN201610322284 A CN 201610322284A CN 106022245 A CN106022245 A CN 106022245A
Authority
CN
China
Prior art keywords
algorithm
parallel processing
remote sensing
data
satellite data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610322284.5A
Other languages
Chinese (zh)
Other versions
CN106022245B (en
Inventor
曹宇
王峰
祝令亚
孙业超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Center for Resource Satellite Data and Applications CRESDA
Original Assignee
China Center for Resource Satellite Data and Applications CRESDA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Center for Resource Satellite Data and Applications CRESDA filed Critical China Center for Resource Satellite Data and Applications CRESDA
Priority to CN201610322284.5A priority Critical patent/CN106022245B/en
Publication of CN106022245A publication Critical patent/CN106022245A/en
Application granted granted Critical
Publication of CN106022245B publication Critical patent/CN106022245B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/10Terrestrial scenes
    • G06V20/13Satellite images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques

Abstract

Provided is a multi-source remote sensing satellite data parallel processing system and method based on algorithm classification. The system and method support new algorithm register, store and manage the registered algorithm, select one or more algorithms from a algorithm register module, push remote sensing satellite data required to be processed and the selected algorithms to a parallel processing node to be processed; the parallel processing nodes simultaneously parallel compute the remote sensing satellite data required to be processed according to the algorithms received by nodes and an executing order; monitors the execution of the algorithms pushed to the parallel processing nodes; and files and stores processed results acquired by the parallel processing nodes. The system and the method support various remote sensing algorithm and multi-source remote sensing data, may simultaneously respond to and execute a multi-source remote sensing data processing request in a distributed cluster, solves multi-source remote sensing satellite data multi-algorithm comprehensive processing, parallel computing, and distributed storage, and achieve a multi-source remote sensing satellite data parallel processing effect.

Description

A kind of multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification and method
Technical field
The present invention relates to a kind of multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification and side Method, belongs to satellite data and processes and application.
Background technology
Satellite data processes and refers to that the initial data passed down by satellite becomes corresponding image through processing, And from satellite data, extract various information, global, high dynamic, the company having by satellite data Continuous property, round-the-clock, round-the-clock, enriched data acquisition feature, be widely used in as agricultural, forestry, water The professional fields such as profit, mapping, traffic, meteorology, ocean provide data support.Its object is to can be Under limited hardware condition, stablize as far as possible, particular data Processing Algorithm is performed quickly, produce high-quality The satellite data product of amount.
Through development for many years, China's satellite information acquisition, process and application technology achieve significantly one-tenth Achievement, has gradually formed meteorology, resource, ocean, environment disaster reduction four big civil remote sensing satellite series.? Under the promotion of the large-scale problems such as 863 Program, high score are special, China's remote sensing application research enter maximization, Rapid developmental stage.Meanwhile, various satellite platforms and sensor device no matter kind, quantity is also It is that quality is all constantly promoting.The data of China's satellite remote sensing ground station reception already more than PB level at present, And will increase with the speed more than 10TB/ days.The data volume that satellite remote sensing date is huge, needs by sea Amount date storage method carries out filing storage.Magnanimity not only includes jumbo data storage capacities, also wraps Include large-scale data throughput capabilities.Along with the growth of portfolio, memory data output and storage performance rapidly Increase, also require that method has high dynamic scalable performance, it is to avoid operation system is made by storage dilatation Growth time interrupts.On the other hand, the satellite application for the purpose of monitoring, calamity emergency etc. is for timeliness Property demand more and more higher, to satellite data process bring new challenge.In the face of the mass data received, Possess efficient data-handling capacity, could meet towards agriculture feelings, the condition of a fire, the condition of a disaster, environment, meteorology Etc. the application demand of conglomerate, and due to processing equipment and the restriction of traditional technology method, still have a large amount of Data the most effectively processed and utilized, general 50% to 90% data are in idle or half idle State, for satellite resource and storage resource significant wastage.Additionally, the cluster parallel processing of routine solves Scheme to ensure that the resource utilization of the highest calculating demand needs the configuration redundant computation more than a times and deposits Storage resource, causes the period major part resource in little calculating demand to be in no-load running state.
Summary of the invention
Present invention solves the technical problem that for: overcome prior art not enough, it is provided that a kind of based on algorithm classification Multi-source remote sensing satellite data parallel processing system (PPS) and method, to solve in prior art, remote sensing satellite number Satellite data kind, satellite data Processing Algorithm and the sea become increasingly complex cannot be mated according to processing system Amount data filing needs with storing, and restriction satellite data integrated treatment is sharp with the autgmentability difference of application and resource By the low defect of rate, it is achieved the satellite data of coupling magnanimity, complicated satellite data Processing Algorithm, it is provided that The ability of the fast parallel calculating of satellite data high-performance.
The technical scheme that the present invention provides is: a kind of multi-source remote sensing satellite data based on algorithm classification is parallel Processing system and method, including algorithm Registering modules, algorithm pushing module, parallel processing module, task Monitoring module, data filing module;
Algorithm Registering modules, supports new algorithm registration, the algorithm registered is stored and managed.Calculate Method Registering modules, supports new algorithm registration, the algorithm registered is stored and managed.Described new calculation All parameters that method mainly performs needed for process should not comprise man-machine interaction, and algorithm execution can be held at algorithm Determine before row;
Algorithm pushing module, according to the needs of task, choose from algorithm Registering modules one of needs or Polyalgorithm, when there being polyalgorithm, it is possible to arranges the execution sequence of polyalgorithm, according to needing Remote sensing satellite data to be processed arrive the migration amount of each parallel processing node, multiple parallel processings are saved Point arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing satellite data move Several nodes that shifting amount is minimum, if these nodes have to meet carries out data process according to selected algorithm Resource, then will need remote sensing satellite data to be processed and the algorithm chosen to deliver to these nodes;If selecting Make that several nodes of remote sensing satellite Data Migration amount minimum have node not have to enter according to selected algorithm The resource that row data process, then postpone and choose having according to choosing beyond these nodes that migration amount is minimum Fixed algorithm carries out the node of the resource requirement of data process, does not have to enter according to selected algorithm to replace The node of the resource requirement that row data process, it would be desirable to the remote sensing satellite data of process and the algorithm chosen send To these nodes;
The algorithm that multiple parallel processing nodes in parallel processing module receive according to each node simultaneously, presses According to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processing nodes obtain Multiple result of calculations also store, and multiple parallel processing nodes can be according to the needs of task, again by multiple Result of calculation double counting uses;
Mission Monitor module, enters the implementation status of the one or more algorithms delivering to each parallel processing node Row monitoring, when task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module energy Enough notice tasks need the next algorithm performed to carry out data process, until task needs all of execution Algorithm terminates.
Data filing module, the result obtaining each parallel processing node carries out filing storage, energy Enough set up the corresponding relation of result and the metadata of result, tie with processing according to this result The corresponding relation of the metadata of fruit, it is possible to transfer the result of needs.
Mission Monitor module can monitor the resource behaviour in service of parallel processing node and record performed The execution time of algorithm.
Described parallel processing node is multiple, and described result splits into the original of multiple 64MB Block, and obtain replicating blocks of files by the blocks of files duplication of the plurality of 64MB, by original block and duplication literary composition Part block is uniformly stored in all parallel processing nodes, and replicates blocks of files and original block not with in the lump Row processes in node.
Algorithm Registering modules, when new algorithm is registered, according to the interface specification set, utilizes XML language Write the parameter list that algorithm needs.
A kind of multi-source remote sensing satellite data method for parallel processing, comprises the following steps that
(1) algorithm Registering modules, supports new algorithm registration, the algorithm registered is stored and managed Reason;
(2) algorithm pushing module, according to the needs of task, selects from step (1) algorithm Registering modules Take one or more algorithms of needs, when there being polyalgorithm, it is possible to the execution sequence of polyalgorithm is entered Row arrangement, the remote sensing satellite data processed as required arrive the migration amount of each parallel processing node, will Multiple parallel processing nodes arrange, and according to the quantity of the parallel processing node that task needs, select to make Several nodes that remote sensing satellite Data Migration amount is minimum, meet according to selected algorithm if these nodes have Carry out the resource of data process, then remote sensing satellite data to be processed and the algorithm chosen will be needed to deliver to these Node;If making of selecting the minimum several nodes of remote sensing satellite Data Migration amount have node do not have according to Selected algorithm carries out the resource of data process, then postpone and choose beyond these nodes that migration amount is minimum The node with the resource requirement carrying out data process according to selected algorithm, with replace do not have according to Selected algorithm carries out the node of the resource requirement of data process, it would be desirable to the remote sensing satellite data of process and The algorithm chosen delivers to these nodes;
(3) calculation that the multiple parallel processing nodes in parallel processing module receive according to each node simultaneously Method, according to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processings save Point obtains multiple result of calculation, and multiple parallel processing nodes can be according to the needs of task, again by multiple Result of calculation double counting uses;
(4) Mission Monitor module, the execution to the one or more algorithms delivering to each parallel processing node Situation is monitored, when task needs to perform polyalgorithm, after an algorithm completes, and Mission Monitor Module can notify that task needs the next algorithm performed to carry out data process, until task needs to perform All algorithms terminate.
(5) data filing module, the result obtaining each parallel processing node carries out filing and deposits Storage, it is possible to set up the corresponding relation of the metadata of result and result, according to this result with The corresponding relation of the metadata of result, it is possible to transfer the result of needs.
Present invention advantage compared with prior art is:
(1) present invention processes remote sensing satellite data processing algorithm is pushed to different process nodes Before, also include: judge aforementioned remote sensing satellite data processing algorithm desired data file storage location, and select Select the node that Data Migration amount is minimum, algorithm is migrated to the node that aforementioned Data Migration amount is minimum;
(2) in the satellite data parallel processing module of the present invention: the executed in parallel stream to described production task Journey is managed and monitors, and resource situation and history by monitoring calculating node perform to be recorded as locating parallel Reason strategy provides foundation;
(3) in the satellite data parallel processing module of the present invention: the result of described production task uses distribution Formula method is stored in all nodes;
(4) distributed storage method of the present invention, also includes: the process result pigeonholing of described production task While storage, set up the index of metadata and data according to default rule;
(5) the algorithm Registering modules of the present invention, also includes: the algorithm interface specification of standard, for meeting The new algorithm of interface specification, it is possible to dynamically include in and participate in described remote sensing satellite data in parallel processing flow process The execution of Processing Algorithm.
(6) present invention provide multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification and In method, it is possible to respond simultaneously and perform the multiple place of multi-source Remote Sensing Data data on distributed computer cluster Adjustment method is asked, and is adaptive selected the node calculating performing operation according to algorithm and data deposit position Machine.Compared with tradition list satellite form processing method, it is possible to make full use of multiformity and the process of satellite data The durability of algorithm, decreases and calculates the migration of data between node.This programme can support all kinds of remote sensing Algorithm and multi-source Remote Sensing Data data, it is possible to respond on distributed type assemblies simultaneously and perform at multi-source Remote Sensing Data data Reason request, reduces the data throughput bottleneck during multi-source satellite data processes, improves production efficiency, and solution is many Source remote sensing satellite data many algorithm synthesis process, parallel computation and the problem of distribution storage, reach multi-source distant The effect of sense satellite data parallel processing.
Accompanying drawing explanation
The multi-source satellite data parallel processing system (PPS) block schematic illustration that Fig. 1 provides for the present invention;
The multi-source satellite data method for parallel processing hierarchical chart that Fig. 2 provides for the present invention;
The multi-source satellite data method for parallel processing data management hierarchical chart that Fig. 3 provides for the present invention;
The marine oil overflow monitoring that Fig. 4 provides for the present invention performs schematic diagram with simulation algorithm.
Detailed description of the invention
The basic ideas of the present invention are: the invention provides a kind of multi-source remote sensing satellite based on algorithm classification Parallel data processing system and method, supports new algorithm registration, the algorithm registered is stored and managed Reason;According to the needs of task, from algorithm Registering modules, choose one or more algorithm, it would be desirable to process Remote sensing satellite data and the algorithm chosen be pushed to parallel processing node and process;Multiple parallel processings The algorithm that node receives according to each node simultaneously, according to execution sequence, to needing remote sensing satellite to be processed Data parallel;The implementation status delivering to each parallel processing node algorithm is monitored;To each also The result that row process node obtains carries out filing storage.This programme supports all kinds of remote sensing algorithms and multi-source Remotely-sensed data, it is possible to respond on distributed type assemblies simultaneously and perform multi-source Remote Sensing Data data process request, solving Certainly multi-source remote sensing satellite data many algorithm synthesis process, parallel computation and the problem of distribution storage, reach many The effect of source remote sensing satellite parallel data processing.
The present invention is described in detail with specific embodiment below in conjunction with the accompanying drawings.
Embodiment one:
For in prior art, remote sensing satellite data handling system cannot mate the satellite number become increasingly complex Filing according to kind, satellite data Processing Algorithm and mass data and need with storage, restriction satellite data is combined Conjunction processes the defect low with the autgmentability of application difference and resource utilization, and the embodiment of the present application realizes coupling sea The satellite data of amount, complicated satellite data Processing Algorithm, it is provided that the fast parallel meter of satellite data high-performance The ability calculated.
In conjunction with system framework schematic diagram described in accompanying drawing 1, the present embodiment is described further, the method bag Containing following steps:
Algorithm Registering modules 101, supports new algorithm registration, the algorithm registered is stored and managed. Described new algorithm mainly performs process should not comprise man-machine interaction, and algorithm perform needed for all parameters can Determine before algorithm performs.According to the interface specification set, XML language is utilized to write what algorithm needed Parameter list.
Concrete, according to the feature of remote sensing algorithm, when registration algorithm, carried by reading algorithm author The registration interface handed over, including name of product, performs program name, the manufacturing parameter of needs and explanation, for Each answer remote sensing algorithm generate specific production procedure.
Algorithm pushing module 102, according to the needs of task, choose needs from algorithm Registering modules one Or polyalgorithm, when there being polyalgorithm, it is possible to the execution sequence of polyalgorithm is arranged, according to Remote sensing satellite data to be processed are needed to arrive the migration amount of each parallel processing node, by multiple parallel processings Node arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing satellite data Several nodes that migration amount is minimum, if these nodes have to meet carries out data process according to selected algorithm Resource, then remote sensing satellite data to be processed and the algorithm chosen will be needed to deliver to these nodes;If selecting Make the minimum several nodes of remote sensing satellite Data Migration amount have node not have according to selected algorithm Carry out the resource of data process, then postpone choose beyond these nodes that migration amount is minimum have according to Selected algorithm carries out the node of the resource requirement of data process, does not have according to selected algorithm to replace Carry out the node of the resource requirement of data process, it would be desirable to the remote sensing satellite data of process and the algorithm chosen Deliver to these nodes.
Parallel processing module 103, the multiple parallel processing nodes in parallel processing module are simultaneously according to each The algorithm that node receives is according to execution sequence, to needing remote sensing satellite data parallel to be processed, many Individual parallel processing node obtains multiple result of calculation and stores, and multiple parallel processing nodes can be according to task Needs, again by multiple result of calculation double countings use.
Parallel processing module hierarchical chart as described in Figure 2, describes preferred parallel processing module Implementation method, is being obtained by lower floor's distributed file system, parallel processing cluster and flow scheduling scheme After algorithm and cluster running status, according to the strategy set, algorithm is pushed to the highest calculating of priority and saves Point performs, and keeps the tracking to algorithm to obtain execution result information.
Parallel processing module, including model component, MDAC, infrastructure service assembly, business clothes Business assembly, resource component.
First, data and algorithm that platform is provided by model component are converted into data model and algorithm model, By MDAC by these models are operated, it is achieved the access to respective resources;Meanwhile, Infrastructure service assembly use the technology such as persistence framework, IOC container to realize distributed file system and Row processes cluster and the management of flow scheduling scheme;Business service assembly utilizes above-mentioned resource to realize various clothes Being engaged in the service of parallel processing business, wherein, business service assembly has service management function and realizes logarithm According to the dynamic sensing of clustered node position, algorithm place the information execution node to Processing Algorithm according to this Carrying out dynamic assignment, algorithm push function performs to move to algorithm the concrete operations of node, task management Function can be inquired about the algorithm information being carrying out and obtain its execution state, and business service assembly has appoints Business scheduling feature is responsible for starting concrete algorithm and is performed task;Then, workflow component uses business service The flow scheduling related service that assembly provides, starts parallel processing flow process;Finally, resource component pass through Call the operation that workflow component handling process is relevant, other modules are issued parallel processing service.
Mission Monitor module 104, the execution feelings to the one or more algorithms delivering to each parallel processing node Condition is monitored, it is possible to the algorithm that the resource behaviour in service of monitoring parallel processing node and record performed The execution time.When task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module The task that can notify needs the next algorithm performed to carry out data process, until task needs the institute performed Algorithm is had to terminate.
Data filing module 105, the result obtaining each parallel processing node carries out filing storage, The corresponding relation of result and the metadata of result can be set up, according to this result and process The corresponding relation of the metadata of result, it is possible to transfer the result of needs.Archiving process is by described process Result splits into the original block of multiple 64MB, and the blocks of files of the plurality of 64MB is replicated Blocks of files, is uniformly stored in original block and duplication blocks of files in all parallel processing nodes, and replicates Blocks of files and original block be not in same parallel processing node.
Data management hierarchical chart as described in Figure 3.Satellite data and result thereof use distributed Method is stored in whole cluster, coordinates data retrieval, extraction, the method for statistical analysis, can manage The attribute of data go forward side by side row space retrieval, the satellite data being distributed in whole cluster and product data are copied Shellfish, to the position specified, keeps the monitoring to mass data information, thus provides data to prop up for step S101 Support, and ensure that minimum data postpones.
Embodiment two:
Supporting new algorithm registration described in embodiment one algorithm Registering modules 101, the algorithm registered is being entered Row stores and management, according to the interface specification set, utilizes XML language to write the parameter that algorithm needs On the basis of list, present embodiments provide a kind of detailed XML language parameter definition, such as table 1 institute Show, the parameter definition table 1 specific as follows of algorithm:
Table 1 parameter definition content
1. described in parameter definition, [ProductName] label is algorithm title;
2. [ManualParams] label contains each subalgorithm parameter value in need, including literary composition Part type parameter [FileArg], value type parameter [ValueArg];
3. [ModelArgs] label contains subalgorithm all appointments parameter, if algorithm comprises many height Operator, then also comprise [ModelArgs] set of tags of multiple correspondence in parameter definition;
4. the file of the All Files type parameter containing current subalgorithm in [FileArg] label is complete Path;
5. [ValueArg] label contains all numerical value of current subalgorithm or character string type Parameter.
In the present embodiment, the standardized XML markup language parameter list to algorithm is needed is used to carry out Definition, is not related to concrete Parameter File in algorithm invoked procedure and resolves or parameter type judgement, especially It is applicable to remote sensing algorithm process inter-trade, that data source is complicated, there is good versatility, and to future More complicated remote sensing algorithm has extensibility.
Embodiment three:
At needs according to task described in embodiment one algorithm pushing module 102, from algorithm Registering modules On the basis of choosing one or more algorithms of needs, present embodiments provide a kind of for different remote sensing calculations The concrete Selection Strategy of method.For complicated satellite data Processing Algorithm, according to strategy set in advance, Being decomposed into one or more serial subalgorithm having certain trigger mechanism, each subalgorithm can be described as The primitive form of parallel computation maps (Map) and abbreviation (Reduce).Such as, marine oil overflow Emulation and storm tide monitor scheduling algorithm, will be decomposed into a series of subalgorithm chains of certain trigger mechanism;
EOS Soil Water Content inversion algorithm can be considered as only comprising a subalgorithm.
Wherein, for different algorithm types, following decomposition can be had to set tactful:
1. single scape independent process class algorithm: same algorithm repeatedly calls as subalgorithm, concrete with algorithm Being constrained to trigger mechanism, the different nodes being assigned in cluster perform.Algorithm perform result by Repeatedly call result and collect filing.
2. more than scape multidate integrated treatment class algorithm: algorithm is decomposed into the circulation of multiple containment mapping abbreviation Subalgorithm, according to nearby principle and node load, selects Data Migration less and light load Node, specifically constrain in multiple node according to algorithm and perform all subalgorithms, finally simultaneously As MapReduce circulation, whole algorithm is obtained algorithm execution result file.
The most semi-automatic interactive remote teaching: according to algorithm specific features, takes out and is applicable in algorithm locate parallel The part of reason, using this part again as independent algorithm with reference to single scape independent process class algorithm or Many scapes multidate integrated treatment class algorithm policy carries out parallel processing.Mutual part is needed to lead to Cross virtualized mode independent operating.
Present embodiments provide the classification policy for dissimilar remote sensing Processing Algorithm, concrete according to algorithm Execution process and call data the corresponding cluster resource of feature configuration and perform flow process, can tackle more Polymorphic type, the algorithm of more complicated execution process, it is simple to suitably perform flow process, energy for algorithms of different distribution Enough improve execution efficiency and the effect of polynary remote sensing satellite data processing algorithm.
Embodiment four:
In the application, the execution for multi-source remote sensing satellite data parallel processing algorithm is a dynamic call With the process of monitoring, definition when needing continuous acquisition algorithm perform state and register according to algorithm controls to calculate The execution process of method.The algorithm present embodiments providing marine oil overflow monitoring and simulation algorithm calls and data Stream embodiment, as described in Figure 4.
1. oil overflowing remote sense area extraction subalgorithm, inputs oil spilling regional remote sensing data, utilizes categorised decision tree Algorithm, has trained the decision tree classification to remotely-sensed data by classification samples, and then has extracted Oil spill area information.Subalgorithm is suitable for parallel, directly obtains the remotely-sensed data of distributed storage, Parallel processing on reason node throughout.
2. oil spilling analogue simulation subalgorithm, for oil spilling emulation data (comprise emulation oil spilling data and ocean current, The data such as weather), use ECOM model, complete the drift of oil spill events elaioleucite and wind The simulation of change process, it is thus achieved that continuous time section oil spill area information;
3.DDDAS data-driven subalgorithm, first, to the remote sensing obtained in step 1 and step 2 Extract area and do Data Integration with ECOM emulation area, it is thus achieved that more accurate oil spill area, And drawn oil spilling initial condition by neural network algorithm, and then it is the most accurate to carry out subsequent time Oil spilling analogue simulation, and the remote sensing oil spill area combining subsequent time show that quality evaluation is tied Really;
4. dynamic result synthon algorithm, after abovementioned steps 2 and step 3 have performed, by time multiple The oil spilling emulation area of phase is depicted as dynamic GIF image;
5. the oil spill area of quality evaluation subalgorithm, the oil spill area that phantom is drawn and Remotely sensed acquisition Do Overlap Analysis, draw spilled oil simulation precision and diffusion tendency accuracy.Input and overflow for remote sensing Oil extracts area, is output as area coincident ratio and diffusion tendency accuracy.
The said method that the present embodiment is provided is according to the execution feature of marine oil overflow monitoring with simulation algorithm Algorithm decomposed and recombinates, improve utilization rate and the execution efficiency of algorithm calculating resource.
Embodiment five:
In the present invention, all kinds of remote sensing satellite data processing algorithms are tested respectively, are formed as follows Conclusion:
1. adhere to separately described in support matrix 2 remote sensing, agricultural, forestry, water conservancy, mapping, traffic, meteorology, The multi-source remote sensing satellite data Processing Algorithm in Deng Ge field, ocean.
2. improve algorithm execution efficiency, process compared to unit, two conditions processing node Lower algorithm execution efficiency is about 200%, and under 8 node condition, algorithm execution efficiency is about 800%, it is linear with PC cluster capability improving that the present invention is capable of algorithm execution efficiency Promote.
3. the present invention improves resource utilization, it is possible to quick collecting data resource also makes full use of institute There is the disposal ability processing node.
Table 2
Embodiment six:
Corresponding to the system described in above-described embodiment, the present embodiment additionally provides a kind of multi-source remote sensing satellite number According to method for parallel processing, comprise the following steps that
(1) new algorithm registration, stores the algorithm registered and manages;
(2) according to the needs of task, from step (1), one or more algorithms of needs are chosen, when When having polyalgorithm, it is possible to the execution sequence of polyalgorithm is arranged, the remote sensing processed as required Satellite data arrives the migration amount of each parallel processing node, is arranged by multiple parallel processing nodes, According to the quantity of the parallel processing node that task needs, select to make the several of remote sensing satellite Data Migration amount minimum Individual node, if these nodes have meets the resource carrying out data process according to selected algorithm, then need to Remote sensing satellite data to be processed and the algorithm chosen deliver to these nodes;If select makes remote sensing satellite number Data process is carried out according to selected algorithm according to several nodes that migration amount is minimum have node not have Resource, then postpone and choose having beyond these nodes that migration amount is minimum and carry out according to selected algorithm The node of the resource requirement that data process, carries out data process to replace not have according to selected algorithm The node of resource requirement, it would be desirable to the remote sensing satellite data of process and the algorithm chosen deliver to these nodes;
(3) algorithm that multiple parallel processing nodes receive according to each node simultaneously, according to execution sequence, To needing remote sensing satellite data parallel to be processed, multiple parallel processing nodes obtain multiple calculating and tie Really, multiple parallel processing nodes can be according to the needs of task, again by multiple result of calculation double countings Use;
(4) implementation status of the one or more algorithms delivering to each parallel processing node is monitored, When task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module can notify to appoint Business needs the next algorithm performed to carry out data process, until task needs all algorithms knot performed Bundle.
(5) result obtaining each parallel processing node carries out filing storage, it is possible at foundation The corresponding relation of the metadata of reason result and result, according to first number of this result Yu result According to corresponding relation, it is possible to transfer the result of needs.
The present embodiment is the device embodiment of embodiment one, two, three, four correspondence, and its similar part is mutual See, do not repeat them here.
In this specification, each embodiment uses the mode gone forward one by one to describe, and each embodiment stresses Being the difference with other embodiments, between each embodiment, identical similar portion sees mutually.
Described above to the disclosed embodiments, makes professional and technical personnel in the field be capable of or uses The present invention.Multiple amendment to these embodiments will be aobvious and easy for those skilled in the art Seeing, generic principles defined herein can be in the situation without departing from the spirit or scope of the present invention Under, realize in other embodiments.Therefore, the present invention is not intended to be limited to that shown in this article these are excellent Select embodiment, and be to fit to the widest model consistent with principles disclosed herein and features of novelty Enclose.

Claims (5)

1. a multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification, it is characterised in that: Return including algorithm Registering modules, algorithm pushing module, parallel processing module, Mission Monitor module, data Shelves module;
Algorithm Registering modules, supports new algorithm registration, the algorithm registered is stored and managed, institute State all parameters needed for algorithm performs to determine before algorithm performs;
Algorithm pushing module, according to the needs of task, choose from algorithm Registering modules one of needs or Polyalgorithm, when there being polyalgorithm, it is possible to arranges the execution sequence of polyalgorithm, according to needing Remote sensing satellite data to be processed arrive the migration amount of each parallel processing node, multiple parallel processings are saved Point arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing satellite data move Several nodes that shifting amount is minimum, if these nodes have to meet carries out data process according to selected algorithm Resource, then will need remote sensing satellite data to be processed and the algorithm chosen to deliver to these nodes;If selecting Make that several nodes of remote sensing satellite Data Migration amount minimum have node not have to enter according to selected algorithm The resource that row data process, then postpone and choose having according to choosing beyond these nodes that migration amount is minimum Fixed algorithm carries out the node of the resource requirement of data process, does not have to enter according to selected algorithm to replace The node of the resource requirement that row data process, it would be desirable to the remote sensing satellite data of process and the algorithm chosen send To these nodes;
The algorithm that multiple parallel processing nodes in parallel processing module receive according to each node simultaneously, presses According to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processing nodes obtain Multiple result of calculations also store, and multiple parallel processing nodes can be according to the needs of task, again by multiple Result of calculation double counting uses;
Mission Monitor module, enters the implementation status of the one or more algorithms delivering to each parallel processing node Row monitoring, when task needs to perform polyalgorithm, after an algorithm completes, Mission Monitor module energy Enough notice tasks need the next algorithm performed to carry out data process, until task needs all of execution Algorithm terminates;
Data filing module, the result obtaining each parallel processing node carries out filing storage, energy Enough set up the corresponding relation of result and the metadata of result, tie with processing according to this result The corresponding relation of the metadata of fruit, it is possible to transfer the result of needs.
A kind of multi-source remote sensing satellite data based on algorithm classification the most according to claim 1 is parallel Processing system, it is characterised in that: Mission Monitor module can monitor the resource of parallel processing node and use shape The execution time of the algorithm that condition and record performed.
A kind of multi-source remote sensing satellite data based on algorithm classification the most according to claim 1 is parallel Processing system, it is characterised in that: described parallel processing node is multiple, described result is split into The original block of multiple 64MB, and obtain replicating blocks of files by the blocks of files duplication of the plurality of 64MB, By original block and replicate blocks of files be uniformly stored in all parallel processing nodes, and replicate blocks of files and Original block is not in same parallel processing node.
A kind of multi-source remote sensing satellite data based on algorithm classification the most according to claim 1 is parallel Processing system, it is characterised in that: algorithm Registering modules is when new algorithm is registered, according to the interface rule set Model, utilizes XML language to write the parameter list that algorithm needs.
5. a multi-source remote sensing satellite data method for parallel processing based on algorithm classification, it is characterised in that Comprise the following steps that
(1) algorithm Registering modules can support that new algorithm is registered, the algorithm registered is carried out store and Management;
(2) algorithm pushing module is chosen from step (1) algorithm Registering modules according to the needs of task needs The one or more algorithms wanted, when there being polyalgorithm, it is possible to arranges the execution sequence of polyalgorithm Row, the remote sensing satellite data processed as required arrive the migration amount of each parallel processing node, by multiple Parallel processing node arranges, and according to the quantity of the parallel processing node that task needs, selects to make remote sensing Several nodes that satellite data migration amount is minimum, are carried out according to selected algorithm if these nodes have to meet The resource that data process, then will need remote sensing satellite data to be processed and the algorithm chosen to deliver to these joints Point;If the several nodes making remote sensing satellite Data Migration amount minimum selected have node not have according to choosing Fixed algorithm carries out the resource of data process, then postpone and choose beyond these nodes that migration amount is minimum There is the node of the resource requirement carrying out data process according to selected algorithm, do not have according to choosing to replace Fixed algorithm carries out the node of the resource requirement of data process, it would be desirable to the remote sensing satellite data of process and choosing The algorithm taken delivers to these nodes;
(3) calculation that the multiple parallel processing nodes in parallel processing module receive according to each node simultaneously Method, according to execution sequence, to needing remote sensing satellite data parallel to be processed, multiple parallel processings save Point obtains multiple result of calculation, and multiple parallel processing nodes can be according to the needs of task, again by multiple Result of calculation double counting uses;
(4) the execution feelings of the Mission Monitor module one or more algorithms to delivering to each parallel processing node Condition is monitored, when task needs to perform polyalgorithm, after an algorithm completes, and Mission Monitor mould Block can notify that task needs the next algorithm performed to carry out data process, until task needs execution All algorithms terminate;
(5) data filing module carries out filing storage to the result that each parallel processing node obtains, The corresponding relation of result and the metadata of result can be set up, according to this result and process The corresponding relation of the metadata of result, it is possible to transfer the result of needs.
CN201610322284.5A 2016-05-16 2016-05-16 A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification Active CN106022245B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610322284.5A CN106022245B (en) 2016-05-16 2016-05-16 A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610322284.5A CN106022245B (en) 2016-05-16 2016-05-16 A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification

Publications (2)

Publication Number Publication Date
CN106022245A true CN106022245A (en) 2016-10-12
CN106022245B CN106022245B (en) 2019-09-06

Family

ID=57097334

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610322284.5A Active CN106022245B (en) 2016-05-16 2016-05-16 A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification

Country Status (1)

Country Link
CN (1) CN106022245B (en)

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106874067A (en) * 2017-01-24 2017-06-20 华南理工大学 Parallel calculating method, apparatus and system based on lightweight virtual machine
CN107315639A (en) * 2017-05-17 2017-11-03 中国科学院遥感与数字地球研究所 Spectrum remote-sensing image data organization method and system during a kind of high based on cluster
CN107609702A (en) * 2017-09-15 2018-01-19 郑州云海信息技术有限公司 A kind of process meteorological data method and device
CN108345497A (en) * 2018-01-17 2018-07-31 千寻位置网络有限公司 GNSS positions execution method and system, the positioning device of simulation offline
CN108985709A (en) * 2018-06-26 2018-12-11 中国科学院遥感与数字地球研究所 Workflow management method towards more satellite data centers collaboration Remote Sensing Products production
CN109150938A (en) * 2017-06-27 2019-01-04 哈尔滨航天恒星数据系统科技有限公司 Satellite application public service platform based on cloud service
CN109344188A (en) * 2018-09-11 2019-02-15 北京航空航天大学 Remote sensing satellite efficiency artificial intelligence statistical method
CN109558937A (en) * 2017-09-27 2019-04-02 三星电子株式会社 The operating method of nerve network system and nerve network system
CN109960573A (en) * 2018-12-29 2019-07-02 天津南大通用数据技术股份有限公司 A kind of cross-domain calculating task dispatching method and system based on Intellisense
CN111680889A (en) * 2020-05-20 2020-09-18 中国地质大学(武汉) Offshore oil leakage source positioning method and device based on cross entropy
CN111722635A (en) * 2020-06-05 2020-09-29 北京空间飞行器总体设计部 Method for parallel processing tasks of remote sensing satellite and remote sensing satellite system
CN111726592A (en) * 2020-06-30 2020-09-29 北京市商汤科技开发有限公司 Method and apparatus for obtaining architecture of image signal processor
CN112308443A (en) * 2020-11-09 2021-02-02 中国科学院空天信息创新研究院 Batch scheduling method and device for remote sensing information product generation workflow
CN112368995A (en) * 2018-06-21 2021-02-12 西门子股份公司 System for data analysis using local device and cloud computing platform
CN112463739A (en) * 2019-09-09 2021-03-09 山东省计算中心(国家超级计算济南中心) Data processing method and system based on ocean mode ROMS
CN112612617A (en) * 2020-12-30 2021-04-06 东方红卫星移动通信有限公司 Satellite telemetry data processing method and system and constellation state monitoring platform
CN112632113A (en) * 2020-12-31 2021-04-09 北京九章云极科技有限公司 Operator management method and operator management system
WO2021129619A1 (en) * 2019-12-27 2021-07-01 中兴通讯股份有限公司 Detection method and device based on laser radar, and computer readable storage medium
CN113641482A (en) * 2021-08-31 2021-11-12 联通(广东)产业互联网有限公司 AI algorithm off-line scheduling method, system, computer equipment and storage medium
CN114461357A (en) * 2021-12-22 2022-05-10 中国科学院空天信息创新研究院 Remote sensing satellite raw data real-time processing flow scheduling engine
CN114489957A (en) * 2022-04-01 2022-05-13 国家卫星海洋应用中心 Remote sensing satellite data processing method and device and electronic equipment
CN114510297A (en) * 2022-03-31 2022-05-17 国家卫星海洋应用中心 Satellite data reprocessing method and device and electronic equipment
CN117056088A (en) * 2023-10-11 2023-11-14 武汉大学 Multi-mode mapping data distributed parallel computing method and system based on MapReduce
CN117573730A (en) * 2024-01-16 2024-02-20 腾讯科技(深圳)有限公司 Data processing method, apparatus, device, readable storage medium, and program product

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110928692B (en) * 2020-01-23 2020-08-07 飞诺门阵(北京)科技有限公司 Task processing method and device and electronic equipment

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101814029A (en) * 2010-04-20 2010-08-25 中国科学院对地观测与数字地球科学中心 Building method capable of expanding processing function quickly in remote sensing image processing system
US8145677B2 (en) * 2007-03-27 2012-03-27 Faleh Jassem Al-Shameri Automated generation of metadata for mining image and text data
CN104299241A (en) * 2014-10-30 2015-01-21 武汉大学 Remote sensing image significance target detection method and system based on Hadoop
US9152881B2 (en) * 2012-09-13 2015-10-06 Los Alamos National Security, Llc Image fusion using sparse overcomplete feature dictionaries
CN105094984A (en) * 2014-11-25 2015-11-25 航天恒星科技有限公司 Resource scheduling method and system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8145677B2 (en) * 2007-03-27 2012-03-27 Faleh Jassem Al-Shameri Automated generation of metadata for mining image and text data
CN101814029A (en) * 2010-04-20 2010-08-25 中国科学院对地观测与数字地球科学中心 Building method capable of expanding processing function quickly in remote sensing image processing system
CN101814029B (en) * 2010-04-20 2013-11-27 中国科学院对地观测与数字地球科学中心 Building method capable of expanding processing function quickly in remote sensing image processing system
US9152881B2 (en) * 2012-09-13 2015-10-06 Los Alamos National Security, Llc Image fusion using sparse overcomplete feature dictionaries
CN104299241A (en) * 2014-10-30 2015-01-21 武汉大学 Remote sensing image significance target detection method and system based on Hadoop
CN105094984A (en) * 2014-11-25 2015-11-25 航天恒星科技有限公司 Resource scheduling method and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
张树凡 等: "基于云计算的多源遥感数据服务系统研究", 《现代电子技术》 *

Cited By (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106874067B (en) * 2017-01-24 2020-10-02 华南理工大学 Parallel computing method, device and system based on lightweight virtual machine
CN106874067A (en) * 2017-01-24 2017-06-20 华南理工大学 Parallel calculating method, apparatus and system based on lightweight virtual machine
CN107315639A (en) * 2017-05-17 2017-11-03 中国科学院遥感与数字地球研究所 Spectrum remote-sensing image data organization method and system during a kind of high based on cluster
CN109150938A (en) * 2017-06-27 2019-01-04 哈尔滨航天恒星数据系统科技有限公司 Satellite application public service platform based on cloud service
CN107609702A (en) * 2017-09-15 2018-01-19 郑州云海信息技术有限公司 A kind of process meteorological data method and device
CN109558937A (en) * 2017-09-27 2019-04-02 三星电子株式会社 The operating method of nerve network system and nerve network system
CN109558937B (en) * 2017-09-27 2023-11-28 三星电子株式会社 Neural network system and operation method of neural network system
CN108345497A (en) * 2018-01-17 2018-07-31 千寻位置网络有限公司 GNSS positions execution method and system, the positioning device of simulation offline
CN112368995A (en) * 2018-06-21 2021-02-12 西门子股份公司 System for data analysis using local device and cloud computing platform
CN108985709A (en) * 2018-06-26 2018-12-11 中国科学院遥感与数字地球研究所 Workflow management method towards more satellite data centers collaboration Remote Sensing Products production
CN109344188A (en) * 2018-09-11 2019-02-15 北京航空航天大学 Remote sensing satellite efficiency artificial intelligence statistical method
CN109960573A (en) * 2018-12-29 2019-07-02 天津南大通用数据技术股份有限公司 A kind of cross-domain calculating task dispatching method and system based on Intellisense
CN109960573B (en) * 2018-12-29 2021-01-08 天津南大通用数据技术股份有限公司 Cross-domain computing task scheduling method and system based on intelligent perception
CN112463739A (en) * 2019-09-09 2021-03-09 山东省计算中心(国家超级计算济南中心) Data processing method and system based on ocean mode ROMS
WO2021129619A1 (en) * 2019-12-27 2021-07-01 中兴通讯股份有限公司 Detection method and device based on laser radar, and computer readable storage medium
CN111680889A (en) * 2020-05-20 2020-09-18 中国地质大学(武汉) Offshore oil leakage source positioning method and device based on cross entropy
CN111680889B (en) * 2020-05-20 2023-08-18 中国地质大学(武汉) Cross entropy-based offshore oil leakage source positioning method and device
CN111722635A (en) * 2020-06-05 2020-09-29 北京空间飞行器总体设计部 Method for parallel processing tasks of remote sensing satellite and remote sensing satellite system
CN111726592B (en) * 2020-06-30 2022-06-21 北京市商汤科技开发有限公司 Method and apparatus for obtaining architecture of image signal processor
CN111726592A (en) * 2020-06-30 2020-09-29 北京市商汤科技开发有限公司 Method and apparatus for obtaining architecture of image signal processor
CN112308443A (en) * 2020-11-09 2021-02-02 中国科学院空天信息创新研究院 Batch scheduling method and device for remote sensing information product generation workflow
CN112612617B (en) * 2020-12-30 2023-06-20 东方红卫星移动通信有限公司 Satellite telemetry data processing method and system and constellation state monitoring platform
CN112612617A (en) * 2020-12-30 2021-04-06 东方红卫星移动通信有限公司 Satellite telemetry data processing method and system and constellation state monitoring platform
CN112632113A (en) * 2020-12-31 2021-04-09 北京九章云极科技有限公司 Operator management method and operator management system
CN113641482B (en) * 2021-08-31 2024-03-22 联通(广东)产业互联网有限公司 AI algorithm offline scheduling method, system, computer equipment and storage medium
CN113641482A (en) * 2021-08-31 2021-11-12 联通(广东)产业互联网有限公司 AI algorithm off-line scheduling method, system, computer equipment and storage medium
CN114461357A (en) * 2021-12-22 2022-05-10 中国科学院空天信息创新研究院 Remote sensing satellite raw data real-time processing flow scheduling engine
CN114461357B (en) * 2021-12-22 2022-11-11 中国科学院空天信息创新研究院 Remote sensing satellite original data real-time processing flow scheduling system
CN114510297A (en) * 2022-03-31 2022-05-17 国家卫星海洋应用中心 Satellite data reprocessing method and device and electronic equipment
CN114489957A (en) * 2022-04-01 2022-05-13 国家卫星海洋应用中心 Remote sensing satellite data processing method and device and electronic equipment
CN117056088A (en) * 2023-10-11 2023-11-14 武汉大学 Multi-mode mapping data distributed parallel computing method and system based on MapReduce
CN117056088B (en) * 2023-10-11 2024-01-19 武汉大学 Multi-mode mapping data distributed parallel computing method and system based on MapReduce
CN117573730A (en) * 2024-01-16 2024-02-20 腾讯科技(深圳)有限公司 Data processing method, apparatus, device, readable storage medium, and program product
CN117573730B (en) * 2024-01-16 2024-04-05 腾讯科技(深圳)有限公司 Data processing method, apparatus, device, readable storage medium, and program product

Also Published As

Publication number Publication date
CN106022245B (en) 2019-09-06

Similar Documents

Publication Publication Date Title
CN106022245A (en) Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification
CN112115198B (en) Urban remote sensing intelligent service platform
CN103425772B (en) A kind of mass data inquiry method with multidimensional information
CN111680025A (en) Method and system for intelligently assimilating space-time information of multi-source heterogeneous data oriented to natural resources
CN107766402A (en) A kind of building dictionary cloud source of houses big data platform
CN106372114A (en) Big data-based online analytical processing system and method
CN103631922B (en) Extensive Web information extracting method and system based on Hadoop clusters
CN102722355A (en) Workflow mechanism-based concurrent ETL (Extract, Transform and Load) conversion method
CN103631657A (en) Task scheduling algorithm based on MapReduce
CN103605662A (en) Distributed computation frame parameter optimizing method, device and system
CN107247799A (en) Data processing method, system and its modeling method of compatible a variety of big data storages
CN107515952A (en) The method and its system of cloud data storage, parallel computation and real-time retrieval
CN109299298A (en) Construction method, device, application method and the system of image fusion model
CN106202378A (en) The immediate processing method of a kind of streaming meteorological data and system
CN103177035A (en) Data query device and data query method in data base
CN106875320A (en) The efficient visual analysis method of ship aeronautical data under cloud environment
CN106991135A (en) Towards the quick tile generation method of remote sensing image data
CN107944765A (en) Intelligence manufacture production scheduling cooperates with the assessment system and appraisal procedure of management and control ability
CN107463151B (en) A kind of complex surface machining multidimensional knowledge cloud cooperating service method
CN112948123A (en) Spark-based grid hydrological model distributed computing method
CN103345485B (en) A kind of mainframe platform dynamic statement automatic generation method and system
CN110048886A (en) A kind of efficient cloud configuration selection algorithm of big data analysis task
CN115170924A (en) Intelligent interpretation system for air, space and ground big data
Zhou et al. Research on the Internet of Things Platform Design for Agricultural Machinery Operation and Operation Management
CN107122849A (en) Product checking total complete time minimization method based on SparkR

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant