CN106022245B - A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification - Google Patents
A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification Download PDFInfo
- Publication number
- CN106022245B CN106022245B CN201610322284.5A CN201610322284A CN106022245B CN 106022245 B CN106022245 B CN 106022245B CN 201610322284 A CN201610322284 A CN 201610322284A CN 106022245 B CN106022245 B CN 106022245B
- Authority
- CN
- China
- Prior art keywords
- algorithm
- parallel processing
- data
- node
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Astronomy & Astrophysics (AREA)
- Remote Sensing (AREA)
- Multimedia (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification supports new algorithm registration, the algorithm registered is stored and managed；According to the needs of task, one or more algorithms are chosen from algorithm registration module, the algorithm of remote sensing satellite data to be treated and selection is pushed to parallel processing node and is handled；Multiple parallel processing nodes are simultaneously according to the received algorithm of each node, according to execution sequence, to remote sensing satellite data parallel to be treated；To sending the executive condition to each parallel processing node algorithm to be monitored；The processing result obtained to each parallel processing node carries out filing storage.This programme supports all kinds of remote sensing algorithms and multi- source Remote Sensing Data data, multi- source Remote Sensing Data data processing request can be responded and executed simultaneously on distributed type assemblies, it solves the problems, such as the more algorithm synthesis processing of multi-source remote sensing satellite data, parallel computation and distribution storage, achievees the effect that multi-source remote sensing satellite data parallel processing.
Description
Technical field
The present invention relates to a kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification, belongs to and defends
Sing data processing and application field.
Background technique
Satellite data processing refers to becomes corresponding image for the initial data that satellite passes down through processing, and from defending
Various information is extracted in sing data, it is the global, high dynamic that has by satellite data, continuity, round-the-clock, round-the-clock, more
Sample data acquisition feature, is widely used in and provides number for professional domains such as agricultural, forestry, water conservancy, mapping, traffic, meteorology, oceans
According to support.Its object is to can under limited hardware condition, it is as stable as possible, be performed quickly specific data processing calculate
Method produces the satellite data product of high quality.
By the development of many years, China's satellite information is obtained, processing and application technology achieve significant achievement, gradually shape
At meteorology, resource, ocean, the big civil remote sensing satellite series of environment disaster reduction four.In the large-scale project such as 863 Program, high score special project
Promotion under, China's remote sensing application research enters enlargement, rapid developing stage.At the same time, various satellite platforms
No matter type, quantity or quality are all constantly being promoted with sensor device.The data of China's satellite remote sensing ground station reception are current
PB grades are alreadyd exceed, and will be increased with the rate more than 10TB/ days.The huge data volume of satellite remote sensing date, needs by sea
Amount date storage method carries out filing storage.Magnanimity not only includes the data storage capacities of large capacity, further includes large-scale number
According to handling capacity.With the growth of portfolio, data storage capacity and storage performance rapidly increase, and also require method to have high
Dynamic scalable performance avoids storage dilatation from causing to interrupt for a long time to operation system.On the other hand, with monitoring, calamity emergency
Satellite application Deng for the purpose of is higher and higher for the demand of timeliness, brings new challenge to satellite data processing.It faces
Received mass data, will have efficient data-handling capacity, just be able to satisfy towards agriculture feelings, fire behavior, the condition of a disaster, environment, meteorology
Etc. the application demands of conglomeraties still have a large amount of data and due to the limitation of processing equipment and traditional technology method
It to effective processing and utilizes, general 50% to 90% data are in idle or half idle state, for satellite resource and storage
Resource significant wastage.In addition, conventional cluster parallel processing solution will guarantee that the resource utilization of highest calculating demand needs
The redundant computation and storage resource more than one times are configured, causes to be in zero load in the period major part resource of small calculating demand
Operating condition.
Summary of the invention
Present invention solves the technical problem that are as follows: it overcomes the shortage of prior art, it is distant to provide a kind of multi-source based on algorithm classification
Feel satellite data parallel processing system (PPS) and method, to solve in the prior art, remote sensing satellite data processing system can not be matched more
Come more complicated satellite data type, satellite data Processing Algorithm and mass data filing and storage needs, restricts satellite number
According to the integrated treatment defect low with the scalability difference and resource utilization of application, the satellite data, complexity for matching magnanimity are realized
Satellite data Processing Algorithm provides the ability of the fast parallel calculating of satellite data high-performance.
A kind of technical solution provided by the invention are as follows: multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification
And method, including algorithm registration module, algorithm pushing module, parallel processing module, task monitoring module, data filing module；
Algorithm registration module supports new algorithm registration, the algorithm registered is stored and managed.Algorithm registers mould
Block supports new algorithm registration, the algorithm registered is stored and managed.The main implementation procedure of new algorithm should not include
Human-computer interaction, and algorithm executes required all parameters and can determine before algorithm executes；
Algorithm pushing module chooses one or more algorithms of needs according to the needs of task from algorithm registration module,
When there is polyalgorithm, can to polyalgorithm execution sequence arrange, the remote sensing satellite data handled as required arrive
Up to the amount of migration of each parallel processing node, multiple parallel processing nodes are arranged, the parallel processing needed according to task
The quantity of node, selection make the smallest several nodes of remote sensing satellite Data Migration amount, meet if these nodes have according to selected
Algorithm carry out data processing resource, then the algorithm of remote sensing satellite data to be treated and selection is sent to these nodes；
If selection make have in the smallest several nodes of remote sensing satellite Data Migration amount node do not have counted according to selected algorithm
According to the resource of processing, then postpones and choose having according to selected algorithm progress data other than these the smallest nodes of the amount of migration
The node of the required resource of processing, to replace the node for not having the required resource for carrying out data processing according to selected algorithm,
The algorithm of remote sensing satellite data to be treated and selection is sent to these nodes；
Multiple parallel processing nodes in parallel processing module are suitable according to executing simultaneously according to the received algorithm of each node
Sequence, to remote sensing satellite data parallel to be treated, multiple parallel processing nodes obtain multiple calculated results and store, more
Multiple calculated results can be computed repeatedly use again according to the needs of task by a parallel processing node；
Task monitoring module is monitored the executive condition for sending one or more algorithms to each parallel processing node,
When task needs to be implemented polyalgorithm, after the completion of an algorithm, task monitoring module can notify what task needed to be implemented
Next algorithm carries out data processing, until all algorithms that task needs to be implemented terminate.
Data filing module, the processing result obtained to each parallel processing node carry out filing storage, can establish place
The corresponding relationship of the metadata of result and processing result is managed, manages result pass corresponding with the metadata of processing result according to this
System, can transfer the processing result of needs.
The algorithm that task monitoring module can monitor the resource behaviour in service of parallel processing node and record executed is held
The row time.
The parallel processing node be it is multiple, the processing result is split into the original block of multiple 64MB, and should
The blocks of files of multiple 64MB replicates to obtain duplication blocks of files, and original block and duplication blocks of files are uniformly stored in all parallel places
It manages in node, and replicates blocks of files and original block not in same parallel processing node.
Algorithm registration module, according to the interface specification of setting, writes algorithm needs using XML language when new algorithm is registered
Parameter list.
A kind of multi-source remote sensing satellite data method for parallel processing, comprises the following steps that
(1) algorithm registration module supports new algorithm registration, the algorithm registered is stored and managed；
(2) algorithm pushing module, according to the needs of task, one that needs are chosen from step (1) algorithm registration module
Or polyalgorithm, when there is polyalgorithm, can to polyalgorithm execution sequence arrange, the remote sensing handled as required
Satellite data reaches the amount of migration of each parallel processing node, multiple parallel processing nodes is arranged, according to task needs
Parallel processing node quantity, selection makes the smallest several nodes of remote sensing satellite Data Migration amount, if these nodes have it is full
Foot carries out the resource of data processing according to selected algorithm, then by the algorithm of remote sensing satellite data to be treated and selection send to
These nodes；If selection makes to have in the smallest several nodes of remote sensing satellite Data Migration amount node not have according to selected calculation
Method carries out the resource of data processing, then postpones and choose having according to selected algorithm other than these the smallest nodes of the amount of migration
The node of the required resource of data processing is carried out, to replace the required resource for not having and carrying out data processing according to selected algorithm
Node, the algorithm of remote sensing satellite data to be treated and selection is sent to these nodes；
(3) multiple parallel processing nodes in parallel processing module are simultaneously according to the received algorithm of each node, according to holding
Row sequence, to remote sensing satellite data parallel to be treated, multiple parallel processing nodes obtain multiple calculated results, multiple
Multiple calculated results can be computed repeatedly use again according to the needs of task by parallel processing node；
(4) task monitoring module supervises the executive condition for sending one or more algorithms to each parallel processing node
Control, when task needs to be implemented polyalgorithm, after the completion of an algorithm, task monitoring module can notify task to need to be implemented
Next algorithm carry out data processing, until all algorithms for needing to be implemented of task terminate.
(5) data filing module, the processing result obtained to each parallel processing node carry out filing storage, can establish
The corresponding relationship of the metadata of processing result and processing result manages result pass corresponding with the metadata of processing result according to this
System, can transfer the processing result of needs.
The advantages of the present invention over the prior art are that:
(1) present invention also wraps before remote sensing satellite data processing algorithm to be pushed to different processing nodes and is handled
It includes: data file storage location needed for judging aforementioned remote sensing satellite data processing algorithm, and select the smallest section of Data Migration amount
Point migrates algorithm to the smallest node of aforementioned Data Migration amount；
(2) in satellite data parallel processing module of the invention: carrying out pipe to the parallel execution process of the production task
Reason and monitoring are recorded as parallel processing strategy by the resource situation and history execution that monitor calculate node and provide foundation；
(3) in satellite data parallel processing module of the invention: the result of the production task is deposited using distributed method
Storage is in all nodes；
(4) distributed storage method of the invention, further includes: the process result pigeonholing of the production task stores same
When, the index of metadata and data is established according to default rule；
(5) algorithm registration module of the invention, further includes: the algorithm interface specification of standard, for meeting interface specification
New algorithm can dynamically be included in the execution that the remote sensing satellite data processing algorithm is participated in parallel processing process.
(6) in the multi-source remote sensing satellite data parallel processing system (PPS) and method provided by the invention based on algorithm classification, energy
Enough a variety of Processing Algorithms requests for responding and executing simultaneously multi- source Remote Sensing Data data on distributed computer cluster, according to algorithm and
The node computer for executing operation is adaptive selected in data storage position.It, can compared with traditional single satellite form processing method
The diversity of satellite data and the reusability of Processing Algorithm are made full use of, the migration of data between calculate node is reduced.We
Case can support all kinds of remote sensing algorithms and multi- source Remote Sensing Data data, can respond and execute simultaneously multi-source remote sensing on distributed type assemblies
Data processing request reduces the data throughput bottleneck in the processing of multi-source satellite data, improves production efficiency, solves multi-source remote sensing and defends
The problem of more algorithm synthesis processing of sing data, parallel computation and distribution storage, reach multi-source remote sensing satellite data parallel processing
Effect.
Detailed description of the invention
Fig. 1 is multi-source satellite data parallel processing system (PPS) block schematic illustration provided by the invention；
Fig. 2 is multi-source satellite data method for parallel processing hierarchical chart provided by the invention；
Fig. 3 is multi-source satellite data method for parallel processing data management hierarchical chart provided by the invention；
Fig. 4 is marine oil overflow provided by the invention monitoring and simulation algorithm execution schematic diagram.
Specific embodiment
Basic ideas of the invention are as follows: the present invention provides a kind of, and the multi-source remote sensing satellite data based on algorithm classification is parallel
Processing system and method support new algorithm registration, the algorithm registered are stored and managed；According to the needs of task, from
One or more algorithms are chosen in algorithm registration module, and the algorithm of remote sensing satellite data to be treated and selection is pushed to simultaneously
Row processing node is handled；Multiple parallel processing nodes are right according to execution sequence simultaneously according to the received algorithm of each node
Remote sensing satellite data parallel to be treated；To sending the executive condition to each parallel processing node algorithm to be monitored；It is right
The processing result that each parallel processing node obtains carries out filing storage.This programme supports all kinds of remote sensing algorithms and multi-source remote sensing number
According to, multi- source Remote Sensing Data data processing can be responded and executed simultaneously on distributed type assemblies requests, solution multi-source remote sensing satellite data
The problem of more algorithm synthesis processing, parallel computation and distribution storage, achieve the effect that multi-source remote sensing satellite data parallel processing.
The present invention is described in detail with specific embodiment below in conjunction with the accompanying drawings.
Embodiment one:
For in the prior art, remote sensing satellite data processing system can not match the satellite data type to become increasingly complex,
Satellite data Processing Algorithm and mass data filing are needed with storage, restrict the scalability of satellite data integrated treatment and application
The difference defect low with resource utilization, the embodiment of the present application realize the satellite data of matching magnanimity, complicated satellite data processing
Algorithm provides the ability of the fast parallel calculating of satellite data high-performance.
System framework schematic diagram, is described further the present embodiment in conjunction with described in attached drawing 1, and this method includes following step
It is rapid:
Algorithm registration module 101 supports new algorithm registration, the algorithm registered is stored and managed.The new calculation
The main implementation procedure of method should not include human-computer interaction, and algorithm executes required all parameters and can determine before algorithm executes.It presses
According to the interface specification of setting, the parameter list of algorithm needs is write using XML language.
Specifically, the characteristics of according to remote sensing algorithm, is connect in registration algorithm by reading the registration that algorithm author is submitted
Mouthful, including name of product, program name, the manufacturing parameter of needs and explanation are executed, it is specific each to answer remote sensing algorithm to generate
Production procedure.
Algorithm pushing module 102 chooses the one or more calculations needed according to the needs of task from algorithm registration module
Method, when there is polyalgorithm, can to polyalgorithm execution sequence arrange, the remote sensing satellite data handled as required
The amount of migration for reaching each parallel processing node arranges multiple parallel processing nodes, the parallel place needed according to task
The quantity of node is managed, selection makes the smallest several nodes of remote sensing satellite Data Migration amount, meets if these nodes have according to choosing
Fixed algorithm carries out the resource of data processing, then send the algorithm of remote sensing satellite data to be treated and selection to these sections
Point；If selection makes to have in the smallest several nodes of remote sensing satellite Data Migration amount node not have according to selected algorithm progress
The resource of data processing, then postpone choose other than these the smallest nodes of the amount of migration have counted according to selected algorithm
According to the node of the required resource of processing, to replace the section for not having the required resource for carrying out data processing according to selected algorithm
Point send the algorithm of remote sensing satellite data to be treated and selection to these nodes.
Parallel processing module 103, multiple parallel processing nodes in parallel processing module are received according to each node simultaneously
Algorithm, according to execution sequence, to remote sensing satellite data parallel to be treated, multiple parallel processing nodes obtain multiple
Calculated result simultaneously stores, and multiple parallel processing nodes again can compute repeatedly multiple calculated results according to the needs of task
It uses.
Parallel processing module hierarchical chart as described in Figure 2, describes the implementation method of preferred parallel processing module,
After obtaining algorithm and cluster operating status by lower layer's distributed file system, parallel processing cluster and flow scheduling scheme,
The calculate node that algorithm is pushed to highest priority is executed according to the strategy of setting, and keeps the tracking to algorithm to obtain
Obtain execution result information.
Parallel processing module, including model component, Data Access Components, infrastructure service component, business service component, resource
Component.
Firstly, the data that platform provides and algorithm are converted data model and algorithm model by model component, visited by data
Ask that component by operating to these models, realizes the access to respective resources；Meanwhile infrastructure service component long-lasting useization
The technologies such as frame, IOC container realize the management to distributed file system, parallel processing cluster and flow scheduling scheme；Business
Serviced component realizes the various services for serving parallel processing business using above-mentioned resource, wherein business service component has industry
Management function of being engaged in is realized to the dynamic sensing of clustered node position where data and algorithm and information holds Processing Algorithm according to this
Row node carries out dynamic assignment, and algorithm push function executes the concrete operations that algorithm is moved to node, task management functions energy
It enough inquires the algorithm information being carrying out and obtains its execution state, business service component has the function of that task schedule is responsible for starting
Specific algorithm executes task；Then, the flow scheduling related service that workflow component is provided using business service component, starting
Parallel processing process；Finally, being sent out by calling the relevant operation of workflow component process flow other modules by resource component
Cloth parallel processing service.
Task monitoring module 104 supervises the executive condition for sending one or more algorithms to each parallel processing node
Control, the execution time for the algorithm that the resource behaviour in service and record that can monitor parallel processing node executed.When task needs
When executing polyalgorithm, after the completion of an algorithm, task monitoring module can notify next algorithm that task needs to be implemented
Data processing is carried out, until all algorithms that task needs to be implemented terminate.
Data filing module 105, the processing result obtained to each parallel processing node carry out filing storage, can establish
The corresponding relationship of the metadata of processing result and processing result manages result pass corresponding with the metadata of processing result according to this
System, can transfer the processing result of needs.The processing result is split into the original block of multiple 64MB by archiving process, and will
The blocks of files of multiple 64MB replicates to obtain duplication blocks of files, original block and duplication blocks of files is uniformly stored in all parallel
It handles in node, and replicates blocks of files and original block not in same parallel processing node.
Data management hierarchical chart as described in Figure 3.Satellite data and its processing result are deposited using distributed method
Storage in entire cluster, cooperation data retrieval, extraction, statistical analysis method, the attribute of data can be managed and carry out space
The satellite data being distributed in entire cluster and product data are copied to specified position by retrieval, keep believing mass data
The monitoring of breath to provide data supporting for step S101, and guarantees that minimum data postpones.
Embodiment two:
Described in one algorithm registration module 101 of embodiment support new algorithm registration, to the algorithm registered carry out storage and
Management, according to the interface specification of setting, on the basis of the parameter list that algorithm needs are write using XML language, the present embodiment is mentioned
A kind of detailed XML language parameter definition has been supplied, as shown in table 1, the parameter definition of algorithm table 1 specific as follows:
1 parameter definition content of table
1. [ProductName] label described in parameter definition is algorithm title；
2. containing each subalgorithm institute parameter value in need, including file type ginseng in [ManualParams] label
Number [FileArg], value type parameter [ValueArg]；
All specified parameters of subalgorithm are contained in [3. ModelArgs] label, if algorithm includes multiple sub- operators,
It also include multiple corresponding [ModelArgs] set of tags in parameter definition；
4. containing the file fullpath of the All Files type parameter of current subalgorithm in [FileArg] label；
5. containing all numerical value of current subalgorithm or the parameter of character string type in [ValueArg] label.
In the present embodiment, it is defined, is being calculated to the parameter list that algorithm needs using standardized XML markup language
Specific Parameter File parsing or parameter type judgement it are not related in method calling process, especially suitable for inter-trade, data source is multiple
Miscellaneous remote sensing algorithm process has good versatility, and has scalability to the following more complicated remote sensing algorithm.
Embodiment three:
According to the needs of task described in the one algorithm pushing module 102 of embodiment, chooses and need from algorithm registration module
One or more algorithms on the basis of, present embodiments provide a kind of specific Selection Strategy for different remote sensing algorithms.It is right
In complicated satellite data Processing Algorithm, according to preset strategy, being decomposed into one or more has certain trigger mechanism
Serial subalgorithm, each subalgorithm can be described as the citation form of parallel computation --- mapping (Map) and abbreviation
(Reduce).For example, marine oil overflow emulation and storm tide monitoring scheduling algorithm, will be decomposed into a series of sons of certain trigger mechanism
Algorithm chain；
EOS Soil Water Content inversion algorithm can be considered as only comprising a subalgorithm.
Wherein, for different algorithm types, there can be decomposition setting strategy below:
1. single scape independent process class algorithm: same algorithm is repeatedly called as subalgorithm, and touching is specifically constrained to algorithm
Hair mechanism, the different nodes being assigned in cluster execute.Algorithm implementing result collects filing by multiple call result.
2. algorithm: being decomposed into the subalgorithm of multiple containment mapping abbreviation circulations by the multidate integrated treatment class algorithm of scape more than,
According to nearby principle and node load, selects Data Migration less and the node of light load, specifically constrained according to algorithm more
A node is performed simultaneously all subalgorithms, finally recycles to obtain algorithm implementing result as MapReduce using entire algorithm and return
Shelves.
3. semi-automatic interactive remote teaching: according to algorithm specific features, the part for being suitable for parallel processing in algorithm is taken out,
It is used as independent algorithm with reference to single scape independent process class algorithm or more scape multidate integrated treatment class algorithm policies again this part
Carry out parallel processing.Need the part of interaction independent operating by way of virtualization.
The classification policy for different type remote sensing Processing Algorithm is present embodiments provided, was specifically executed according to algorithm
It journey and calls the corresponding cluster resource of feature configuration of data and executes process, more polymorphic type can be coped with, more complicated executed
The algorithm of journey can be improved polynary remote sensing satellite data processing algorithm convenient for executing process for algorithms of different distribution is suitable
Execution efficiency and effect.
Example IV:
In the application, the execution for multi-source remote sensing satellite data parallel processing algorithm is a dynamic call and monitoring
Process, the implementation procedure of definition control algolithm when continuous acquisition algorithm being needed to execute state and register according to algorithm.This implementation
Example provides algorithm calling and data flow embodiment of the marine oil overflow monitoring with simulation algorithm, as described in Figure 4.
1. oil overflowing remote sense area extraction subalgorithm, input oil spilling regional remote sensing data are passed through using categorised decision tree algorithm
The decision tree classification to remotely-sensed data is completed in classification samples training, and then extracts oil spill area information.Subalgorithm is applicable in parallel, directly
The remotely-sensed data for taking distributed storage is obtained, parallel processing on node is throughout managed.
2. oil spilling analogue simulation subalgorithm (includes the emulation number such as oil spilling data and ocean current, weather for oil spilling emulation data
According to), using ECOM model, the simulation to the drift of oil spill events elaioleucite and efflorescence is completed, continuous time period oil spilling is obtained
Area information；
3.DDDAS data-driven subalgorithm, firstly, to obtained Remotely sensed acquisition area and ECOM in step 1 and step 2
Emulation area does Data Integration, obtains more accurate oil spill area, and obtain oil spilling primary condition by neural network algorithm, into
And the more accurate oil spilling analogue simulation of subsequent time is carried out, and the remote sensing oil spill area of subsequent time is combined to obtain quality evaluation
As a result；
4. dynamic result synthesizes subalgorithm, after the completion of abovementioned steps 2 and step 3 execute, the oil spilling of multiple phases is imitated
True face product is depicted as dynamic GIF image；
5. the oil spill area of oil spill area and Remotely sensed acquisition that simulation model obtains is done stacked point by quality evaluation subalgorithm
Analysis, obtains spilled oil simulation precision and diffusion tendency accuracy.Input is that remote sensing oil spilling extracts area, is exported as area coincident ratio
And diffusion tendency accuracy.
The above method provided by the present embodiment according to marine oil overflow monitoring and the execution feature of simulation algorithm to algorithm into
Row decomposes and recombination, improves the execution efficiency of the utilization rate and algorithm to computing resource.
Embodiment five:
In the present invention, all kinds of remote sensing satellite data processing algorithms are tested respectively, formation is such as drawn a conclusion:
1. adhered to separately described in support matrix 2 remote sensing, agricultural, forestry, water conservancy, mapping, traffic, meteorology, the ocean field Deng Ge it is more
Source remote sensing satellite data processing algorithm.
2. improving algorithm execution efficiency, compared to single machine processing, algorithm executes effect under conditions of two processing nodes
Rate is about 200%, and algorithm execution efficiency is about 800% under 8 node conditions, and the present invention can be realized algorithm execution efficiency with collection
Group's computing capability is promoted and is linearly promoted.
3. the present invention improves resource utilization, quick collecting data resource and all processing nodes can be made full use of
Processing capacity.
Table 2
Embodiment six:
Corresponding to system described in above-described embodiment, the present embodiment additionally provides a kind of multi-source remote sensing satellite data and locates parallel
Reason method, comprises the following steps that
(1) new algorithm is registered, and the algorithm registered is stored and managed；
(2) according to the needs of task, from the one or more algorithms for choosing needs in step (1), when there is polyalgorithm,
Can the execution sequence to polyalgorithm arrange, the remote sensing satellite data that handle as required reach each parallel processing section
The amount of migration of point, multiple parallel processing nodes are arranged, and according to the quantity for the parallel processing node that task needs, selection makes
The smallest several nodes of remote sensing satellite Data Migration amount, if these nodes, which have to meet, carries out data processing according to selected algorithm
Resource, then the algorithm of remote sensing satellite data to be treated and selection is sent to these nodes；If selection makes remote sensing satellite
There is node not have the resource for carrying out data processing according to selected algorithm in the smallest several nodes of Data Migration amount, then postpones
Choose the section with the required resource for carrying out data processing according to selected algorithm other than these the smallest nodes of the amount of migration
Point is defended remote sensing to be treated with replacing the node for not having the required resource for carrying out data processing according to selected algorithm
Sing data and the algorithm of selection are sent to these nodes；
(3) multiple parallel processing nodes are simultaneously according to the received algorithm of each node, according to execution sequence, to needing to handle
Remote sensing satellite data parallel, multiple parallel processing nodes obtain multiple calculated results, and multiple parallel processing nodes can
According to the needs of task, multiple calculated results are computed repeatedly into use again；
(4) executive condition for sending one or more algorithms to each parallel processing node is monitored, when task needs
When executing polyalgorithm, after the completion of an algorithm, task monitoring module can notify next algorithm that task needs to be implemented
Data processing is carried out, until all algorithms that task needs to be implemented terminate.
(5) processing result obtained to each parallel processing node carries out filing storage, can establish processing result and place
The corresponding relationship of the metadata of result is managed, the corresponding relationship of the metadata of result and processing result is managed according to this, can transfer
The processing result needed.
The present embodiment be the corresponding Installation practice of embodiment one, two, three, four, similar place cross-reference,
This is repeated no more.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with other
The difference of embodiment, the same or similar parts in each embodiment may refer to each other.
The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention.
Various modifications to these embodiments will be readily apparent to those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, of the invention
These preferred embodiments shown in this article are not intended to be limited to, and are to fit to and the principles and novel features disclosed herein
Consistent widest scope.
Claims (4)
1. a kind of multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification, the satellite data of realization matching magnanimity,
Complicated satellite data Processing Algorithm, provides the ability of the fast parallel calculating of satellite data high-performance, it is characterised in that: including calculating
Method registration module, algorithm pushing module, parallel processing module, task monitoring module, data filing module；
Algorithm registration module supports new algorithm registration, the algorithm registered is stored and managed, needed for the algorithm executes
All parameters algorithm execute before determine；The main implementation procedure of new algorithm should not include human-computer interaction；For complexity
Satellite data Processing Algorithm is decomposed into one or more series for having certain trigger mechanism and calculates according to preset strategy
Method, each subalgorithm can be described as the citation form of parallel computation --- mapping (Map) and abbreviation (Reduce)；For not
Same algorithm types can have decomposition setting strategy below:
(1) single scape independent process class algorithm: same algorithm is repeatedly called as subalgorithm, and triggering machine is specifically constrained to algorithm
System, the different nodes being assigned in cluster execute；Algorithm implementing result collects filing by multiple call result；
(2) algorithm more scape multidate integrated treatment class algorithms: is decomposed into the subalgorithm of multiple containment mapping abbreviation circulations, foundation
Nearby principle and node load select Data Migration less and the node of light load, multiple sections are specifically constrained according to algorithm
Point is performed simultaneously all subalgorithms, finally recycles to obtain algorithm implementing result as MapReduce using entire algorithm and file；
(3) semi-automatic interactive remote teaching: according to algorithm specific features, taking out the part for being suitable for parallel processing in algorithm, will
This part be used as again independent algorithm with reference to single scape independent process class algorithm or more scape multidate integrated treatment class algorithm policies into
Row parallel processing；Need the part of interaction independent operating by way of virtualization；
Algorithm pushing module chooses one or more algorithms of needs according to the needs of task from algorithm registration module, when having
When polyalgorithm, can the execution sequence to polyalgorithm arrange, the remote sensing satellite data that handle as required reach every
The amount of migration of a parallel processing node arranges multiple parallel processing nodes, the parallel processing node needed according to task
Quantity, selection make the smallest several nodes of remote sensing satellite Data Migration amount, if these nodes have meet according to selected calculation
Method carries out the resource of data processing, then send the algorithm of remote sensing satellite data to be treated and selection to these nodes；If choosing
That selects makes to have in the smallest several nodes of remote sensing satellite Data Migration amount node not have according at selected algorithm progress data
The resource of reason then postpones and chooses having according to selected algorithm progress data processing other than these the smallest nodes of the amount of migration
The node of required resource need to replace the node for not having the required resource for carrying out data processing according to selected algorithm
Remote sensing satellite data to be processed and the algorithm of selection are sent to these nodes；
Multiple parallel processing nodes in parallel processing module are simultaneously according to the received algorithm of each node, according to execution sequence,
To remote sensing satellite data parallel to be treated, multiple parallel processing nodes obtain multiple calculated results and store, multiple
Multiple calculated results can be computed repeatedly use again according to the needs of task by parallel processing node；
Parallel processing module, including model component, Data Access Components, infrastructure service component, business service component, resource group
Part；
Firstly, the data that platform provides and algorithm are converted data model and algorithm model by model component, by data access group
Part realizes the access to respective resources by operating to these models；Meanwhile infrastructure service component long-lasting useization frame
The technologies such as frame, IOC container realize the management to distributed file system, parallel processing cluster and flow scheduling scheme；Business clothes
Business component realizes the various services for serving parallel processing business using above-mentioned resource, wherein business service component has business
Management function is realized to the dynamic sensing of clustered node position and execution of the information to Processing Algorithm according to this where data and algorithm
Node carries out dynamic assignment, and algorithm push function executes the concrete operations that algorithm is moved to node, and task management functions can
It inquires the algorithm information being carrying out and obtains its execution state, business service component has the function of that task schedule is responsible for starting tool
The algorithm of body executes task；Then, the flow scheduling related service that workflow component is provided using business service component, starting is simultaneously
Row process flow；Finally, being issued by calling the relevant operation of workflow component process flow to other modules by resource component
Parallel processing service；
Task monitoring module is monitored the executive condition for sending one or more algorithms to each parallel processing node, when appoint
When business needs to be implemented polyalgorithm, after the completion of an algorithm, task monitoring module can notify task to need to be implemented next
A algorithm carries out data processing, until all algorithms that task needs to be implemented terminate；
Data filing module, the processing result obtained to each parallel processing node carry out filing storage, can establish processing knot
The corresponding relationship of the metadata of fruit and processing result manages the corresponding relationship of the metadata of result and processing result, energy according to this
Enough transfer the processing result of needs；
Parallel processing node be it is multiple, the processing result splits into the original block of multiple 64MB, and by multiple 64MB
Blocks of files replicate to obtain duplication blocks of files, original block and duplication blocks of files are uniformly stored in all parallel processing nodes
In, and blocks of files and original block are replicated not in same parallel processing node；
Before remote sensing satellite data processing algorithm to be pushed to different processing nodes and is handled, further includes: judge aforementioned distant
Data file storage location needed for feeling satellite data Processing Algorithm, and the smallest node of Data Migration amount is selected, forward by algorithm
State the smallest node migration of Data Migration amount；
In satellite data parallel processing module: the parallel execution process of production task being managed and monitored, is counted by monitoring
Resource situation and the history execution of operator node are recorded as parallel processing strategy and provide foundation；
In satellite data parallel processing module: the result of production task is stored in all nodes using distributed method；
Distributed storage method, further includes: while the process result pigeonholing storage of production task, established according to default rule
The index of metadata and data；
Algorithm registration module, further includes: the algorithm interface specification of standard being capable of dynamic for meeting the new algorithm of interface specification
It is included in the execution that the remote sensing satellite data processing algorithm is participated in parallel processing process in ground.
2. a kind of multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification according to claim 1, special
Sign is: the execution for the algorithm that task monitoring module can monitor the resource behaviour in service of parallel processing node and record executed
Time.
3. a kind of multi-source remote sensing satellite data parallel processing system (PPS) based on algorithm classification according to claim 1, special
Sign is: algorithm registration module, according to the interface specification of setting, writes algorithm needs using XML language when new algorithm is registered
Parameter list.
4. a kind of multi-source remote sensing satellite data method for parallel processing based on algorithm classification, the satellite data of realization matching magnanimity,
Complicated satellite data Processing Algorithm, provides the ability of the fast parallel calculating of satellite data high-performance, which is characterized in that including step
It is rapid as follows:
(1) algorithm registration module can support new algorithm to register, and the algorithm registered is stored and managed；Algorithm registers mould
Block supports new algorithm registration, the algorithm registered is stored and managed；The main implementation procedure of new algorithm should not include
Human-computer interaction；All parameters needed for the algorithm executes determine before algorithm executes；Algorithm registration module further include: standard
Algorithm interface specification, for meeting the new algorithm of interface specification, can dynamically be included in parallel processing process participate in it is described distant
Feel the execution of satellite data Processing Algorithm；Complicated satellite data Processing Algorithm is decomposed into according to preset strategy
One or more has the serial subalgorithm of certain trigger mechanism, each subalgorithm can be described as the fundamental form of parallel computation
Formula --- mapping (Map) and abbreviation (Reduce)；For different algorithm types, there can be decomposition setting strategy below:
(1.1) single scape independent process class algorithm: same algorithm is repeatedly called as subalgorithm, and triggering is specifically constrained to algorithm
Mechanism, the different nodes being assigned in cluster execute；Algorithm implementing result collects filing by multiple call result；
(1.2) algorithm: being decomposed into the subalgorithm of multiple containment mapping abbreviation circulations by more scape multidate integrated treatment class algorithms, according to
According to nearby principle and node load, selects Data Migration less and the node of light load, specifically constrained according to algorithm multiple
Node is performed simultaneously all subalgorithms, finally recycles to obtain algorithm implementing result as MapReduce using entire algorithm and return
Shelves；
(1.3) semi-automatic interactive remote teaching: according to algorithm specific features, taking out the part for being suitable for parallel processing in algorithm,
It is used as independent algorithm with reference to single scape independent process class algorithm or more scape multidate integrated treatment class algorithm policies again this part
Carry out parallel processing；Need the part of interaction independent operating by way of virtualization；
(2) algorithm pushing module chooses the one or more of needs according to the needs of task from step (1) algorithm registration module
Algorithm, when there is polyalgorithm, can to polyalgorithm execution sequence arrange, the remote sensing satellite number handled as required
According to the amount of migration for reaching each parallel processing node, multiple parallel processing nodes are arranged, are needed according to task parallel
Handle node quantity, selection make the smallest several nodes of remote sensing satellite Data Migration amount, if these nodes have meet according to
Selected algorithm carries out the resource of data processing, then send the algorithm of remote sensing satellite data to be treated and selection to these sections
Point；Before remote sensing satellite data processing algorithm to be pushed to different processing nodes and is handled, further includes: judge aforementioned remote sensing
Data file storage location needed for satellite data Processing Algorithm, and select the smallest node of Data Migration amount, by algorithm to aforementioned
The smallest node migration of Data Migration amount；If selection make have node not in the smallest several nodes of remote sensing satellite Data Migration amount
With the resource for carrying out data processing according to selected algorithm, then the tool chosen other than these the smallest nodes of the amount of migration of postponing
There is the node for the required resource for carrying out data processing according to selected algorithm, does not have with replacement and counted according to selected algorithm
According to the node of the required resource of processing, the algorithm of remote sensing satellite data to be treated and selection is sent to these nodes；
(3) in parallel processing module: the parallel execution process of production task being managed and monitored, by monitoring calculate node
Resource situation and history execution be recorded as parallel processing strategy provide foundation；The result of production task is deposited using distributed method
Storage is in all nodes；Distributed storage method, further includes: while the process result pigeonholing storage of production task, according to pre-
If rule establish the indexes of metadata and data；Multiple parallel processing nodes in parallel processing module are simultaneously according to each section
The received algorithm of point, according to execution sequence, to remote sensing satellite data parallel to be treated, multiple parallel processing nodes are obtained
To multiple calculated results, multiple parallel processing nodes again can be computed repeatedly multiple calculated results according to the needs of task
It uses；Parallel processing module, including model component, Data Access Components, infrastructure service component, business service component, resource group
Part；Firstly, the data that platform provides and algorithm are converted data model and algorithm model by model component, by Data Access Components
By operating to these models, the access to respective resources is realized；Meanwhile infrastructure service component long-lasting useization frame,
IOC container technique realizes the management to distributed file system, parallel processing cluster and flow scheduling scheme；Business service component
The various services for serving parallel processing business are realized using above-mentioned resource, wherein business service component has service management function
Be able to achieve to the dynamic sensing of clustered node position where data and algorithm and according to this information to the execution node of Processing Algorithm into
Mobile state is assigned, and algorithm push function executes the concrete operations that algorithm is moved to node, and task management functions can be inquired just
Execution algorithm information and obtain its execution state, business service component has the function of that task schedule is responsible for starting specific calculation
Method executes task；Then, the flow scheduling related service that workflow component is provided using business service component starts parallel processing
Process；Finally, issuing parallel place to other modules by resource component by calling the relevant operation of workflow component process flow
Reason service；
(4) task monitoring module is monitored the executive condition for sending one or more algorithms to each parallel processing node, when
When task needs to be implemented polyalgorithm, after the completion of an algorithm, task monitoring module can notify task to need to be implemented down
One algorithm carries out data processing, until all algorithms that task needs to be implemented terminate；
(5) processing result that data filing module obtains each parallel processing node carries out filing storage, can establish processing
As a result with the corresponding relationship of the metadata of processing result, the corresponding relationship of the metadata of result and processing result is managed according to this,
The processing result of needs can be transferred.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610322284.5A CN106022245B (en) | 2016-05-16 | 2016-05-16 | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610322284.5A CN106022245B (en) | 2016-05-16 | 2016-05-16 | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106022245A CN106022245A (en) | 2016-10-12 |
| CN106022245B true CN106022245B (en) | 2019-09-06 |
Family
ID=57097334
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610322284.5A Active CN106022245B (en) | 2016-05-16 | 2016-05-16 | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106022245B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928692A (en) * | 2020-01-23 | 2020-03-27 | 飞诺门阵(北京)科技有限公司 | Task processing method and device and electronic equipment |
Families Citing this family (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106874067B (en) * | 2017-01-24 | 2020-10-02 | 华南理工大学 | Parallel computing method, device and system based on lightweight virtual machine |
| CN107315639A (en) * | 2017-05-17 | 2017-11-03 | 中国科学院遥感与数字地球研究所 | Spectrum remote-sensing image data organization method and system during a kind of high based on cluster |
| CN109150938A (en) * | 2017-06-27 | 2019-01-04 | 哈尔滨航天恒星数据系统科技有限公司 | Satellite application public service platform based on cloud service |
| CN107609702A (en) * | 2017-09-15 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of process meteorological data method and device |
| KR102610820B1 (en) * | 2017-09-27 | 2023-12-06 | 삼성전자주식회사 | Neural network system, and Operating method of neural network system |
| CN108345497A (en) * | 2018-01-17 | 2018-07-31 | 千寻位置网络有限公司 | GNSS positions execution method and system, the positioning device of simulation offline |
| EP3588910A1 (en) | 2018-06-21 | 2020-01-01 | Siemens Aktiengesellschaft | System for data analytics using a local device and a cloud computing platform |
| CN108985709A (en) * | 2018-06-26 | 2018-12-11 | 中国科学院遥感与数字地球研究所 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
| CN109344188B (en) * | 2018-09-11 | 2022-10-04 | 北京航空航天大学 | Artificial intelligence statistical method for remote sensing satellite efficiency |
| CN109960573B (en) * | 2018-12-29 | 2021-01-08 | 天津南大通用数据技术股份有限公司 | Cross-domain computing task scheduling method and system based on intelligent perception |
| CN112463739A (en) * | 2019-09-09 | 2021-03-09 | 山东省计算中心（国家超级计算济南中心） | Data processing method and system based on ocean mode ROMS |
| CN113050993A (en) | 2019-12-27 | 2021-06-29 | 中兴通讯股份有限公司 | Laser radar-based detection method and device and computer-readable storage medium |
| CN111680889B (en) * | 2020-05-20 | 2023-08-18 | 中国地质大学（武汉） | Cross entropy-based offshore oil leakage source positioning method and device |
| CN111722635B (en) * | 2020-06-05 | 2023-06-06 | 北京空间飞行器总体设计部 | Method for processing tasks in parallel by remote sensing satellite and remote sensing satellite system |
| CN111726592B (en) * | 2020-06-30 | 2022-06-21 | 北京市商汤科技开发有限公司 | Method and apparatus for obtaining architecture of image signal processor |
| CN112308443B (en) * | 2020-11-09 | 2024-07-12 | 中国科学院空天信息创新研究院 | Batch scheduling method and device for remote sensing information product generation workflow |
| CN112612617B (en) * | 2020-12-30 | 2023-06-20 | 东方红卫星移动通信有限公司 | Satellite telemetry data processing method and system and constellation state monitoring platform |
| CN112632113B (en) * | 2020-12-31 | 2022-02-11 | 北京九章云极科技有限公司 | Operator management method and operator management system |
| CN113641482B (en) * | 2021-08-31 | 2024-03-22 | 联通(广东)产业互联网有限公司 | AI algorithm offline scheduling method, system, computer equipment and storage medium |
| CN114461357B (en) * | 2021-12-22 | 2022-11-11 | 中国科学院空天信息创新研究院 | Remote sensing satellite original data real-time processing flow scheduling system |
| CN114510297B (en) * | 2022-03-31 | 2022-07-01 | 国家卫星海洋应用中心 | Satellite data reprocessing method and device and electronic equipment |
| CN114489957B (en) * | 2022-04-01 | 2022-07-22 | 国家卫星海洋应用中心 | Remote sensing satellite data processing method and device and electronic equipment |
| CN117056088B (en) * | 2023-10-11 | 2024-01-19 | 武汉大学 | Multi-mode mapping data distributed parallel computing method and system based on MapReduce |
| CN117573730B (en) * | 2024-01-16 | 2024-04-05 | 腾讯科技（深圳）有限公司 | Data processing method, apparatus, device, readable storage medium, and program product |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101814029A (en) * | 2010-04-20 | 2010-08-25 | 中国科学院对地观测与数字地球科学中心 | Building method capable of expanding processing function quickly in remote sensing image processing system |
| US8145677B2 (en) * | 2007-03-27 | 2012-03-27 | Faleh Jassem Al-Shameri | Automated generation of metadata for mining image and text data |
| CN104299241A (en) * | 2014-10-30 | 2015-01-21 | 武汉大学 | Remote sensing image significance target detection method and system based on Hadoop |
| US9152881B2 (en) * | 2012-09-13 | 2015-10-06 | Los Alamos National Security, Llc | Image fusion using sparse overcomplete feature dictionaries |
| CN105094984A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Resource scheduling method and system |
-
2016
- 2016-05-16 CN CN201610322284.5A patent/CN106022245B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8145677B2 (en) * | 2007-03-27 | 2012-03-27 | Faleh Jassem Al-Shameri | Automated generation of metadata for mining image and text data |
| CN101814029A (en) * | 2010-04-20 | 2010-08-25 | 中国科学院对地观测与数字地球科学中心 | Building method capable of expanding processing function quickly in remote sensing image processing system |
| CN101814029B (en) * | 2010-04-20 | 2013-11-27 | 中国科学院对地观测与数字地球科学中心 | Building method capable of expanding processing function quickly in remote sensing image processing system |
| US9152881B2 (en) * | 2012-09-13 | 2015-10-06 | Los Alamos National Security, Llc | Image fusion using sparse overcomplete feature dictionaries |
| CN104299241A (en) * | 2014-10-30 | 2015-01-21 | 武汉大学 | Remote sensing image significance target detection method and system based on Hadoop |
| CN105094984A (en) * | 2014-11-25 | 2015-11-25 | 航天恒星科技有限公司 | Resource scheduling method and system |
Non-Patent Citations (1)
| Title |
|---|
| 基于云计算的多源遥感数据服务系统研究;张树凡 等;《现代电子技术》;20150201;正文第90-94页 |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928692A (en) * | 2020-01-23 | 2020-03-27 | 飞诺门阵(北京)科技有限公司 | Task processing method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106022245A (en) | 2016-10-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106022245B (en) | A kind of multi-source remote sensing satellite data parallel processing system (PPS) and method based on algorithm classification | |
| WO2021213293A1 (en) | Ubiquitous operating system oriented toward group intelligence perception | |
| CN105045820B (en) | Method for processing video image information of high-level data and database system | |
| CN106547882A (en) | A kind of real-time processing method and system of big data of marketing in intelligent grid | |
| CN103631922B (en) | Extensive Web information extracting method and system based on Hadoop clusters | |
| CN110908641B (en) | Visualization-based stream computing platform, method, device and storage medium | |
| CN102567312A (en) | Machine translation method based on distributive parallel computation framework | |
| CN103607466B (en) | A kind of wide-area multi-stage distributed parallel grid analysis method based on cloud computing | |
| CN110740079B (en) | Full link benchmark test system for distributed scheduling system | |
| CN109062780A (en) | The development approach and terminal device of automatic test cases | |
| CN113706099A (en) | Data labeling and deep learning model training and service publishing system | |
| CN109784708A (en) | The cloud service system that the coupling of water industry multi-model calculates | |
| CN108984610A (en) | A kind of method and system based on the offline real-time processing data of big data frame | |
| CN112948123B (en) | Spark-based grid hydrological model distributed computing method | |
| CN109815254A (en) | Cross-region method for scheduling task and system based on big data | |
| CN109087030A (en) | Realize method, General Mobile crowdsourcing server and the system of the crowdsourcing of C2C General Mobile | |
| CN116127899B (en) | Chip design system, method, electronic device, and storage medium | |
| CN107194411A (en) | A kind of SVMs parallel method of improved layering cascade | |
| CN107798059A (en) | A kind of NCO meteorological datas structured storage method and apparatus | |
| Ramesh et al. | Station-level demand prediction for bike-sharing system | |
| CN114693262A (en) | Smart city information grid operating system | |
| CN109376162A (en) | Table data processing method, terminal device and computer readable storage medium | |
| CN107784039A (en) | A kind of data load method, apparatus and system | |
| CN114780742A (en) | Construction and use method of flow scheduling knowledge-graph question-answering system of irrigation area | |
| CN114077492B (en) | Prediction model training and prediction method and system for cloud computing infrastructure resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |