CN104484230B - More satellite data central task stream dispatching algorithms based on nearly data calculating principle - Google Patents
More satellite data central task stream dispatching algorithms based on nearly data calculating principle Download PDFInfo
- Publication number
- CN104484230B CN104484230B CN201410851865.9A CN201410851865A CN104484230B CN 104484230 B CN104484230 B CN 104484230B CN 201410851865 A CN201410851865 A CN 201410851865A CN 104484230 B CN104484230 B CN 104484230B
- Authority
- CN
- China
- Prior art keywords
- data
- algorithm
- resource
- workflow
- nearly
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 119
- 238000000034 method Methods 0.000 claims abstract description 54
- 238000013508 migration Methods 0.000 claims abstract description 28
- 230000005012 migration Effects 0.000 claims abstract description 28
- 230000005540 biological transmission Effects 0.000 claims abstract description 23
- 230000002567 autonomic effect Effects 0.000 claims abstract description 10
- 230000007613 environmental effect Effects 0.000 claims description 33
- 230000008569 process Effects 0.000 claims description 15
- 238000004806 packaging method and process Methods 0.000 claims description 10
- 230000006837 decompression Effects 0.000 claims description 9
- 230000006835 compression Effects 0.000 claims description 7
- 238000007906 compression Methods 0.000 claims description 7
- 238000005538 encapsulation Methods 0.000 claims description 7
- 238000005457 optimization Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 abstract description 18
- 238000012545 processing Methods 0.000 abstract description 16
- 238000010276 construction Methods 0.000 abstract description 7
- 230000008901 benefit Effects 0.000 abstract description 2
- 238000013396 workstream Methods 0.000 abstract description 2
- 238000012546 transfer Methods 0.000 description 5
- 241001269238 Data Species 0.000 description 4
- 230000002035 prolonged effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000010365 information processing Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000009969 flowable effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000004064 recycling Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Radio Relay Systems (AREA)
Abstract
More satellite data central task stream dispatching algorithms based on nearly data calculating principle that the invention discloses a kind of, including step 1: construction work stream executes the nearly data computation model that generalized time minimizes；Step 2: the Virtual Workflow dynamic fixing method based on computing resource processing capacity Yu critical data resource transmission cost；Step 3: the task state PGH that is divided per the group based on Hypergraph Theory optimizes the Virtual Workflow scheduling in step 2；Step 4: Autonomic Migration Framework is carried out to the algorithm between the nearly data computation model of pre-set more satellite data center calculations.The time is always executed the invention has the benefit that can realize and farthest reduce workflow, to greatly improve the workflow schedule efficiency at more satellite data centers, can one-stop remotely-sensed data service of goods mode of the rapid build based on more satellite data centers framework, and for cope with multi-user complicated demand technical support is provided.
Description
Technical field
The present invention relates to one kind towards more satellite data central task stream dispatching algorithms, it particularly relates to which one kind is based on
More satellite data central task stream dispatching algorithms of nearly data calculating principle.
Background technique
With the development of multiple sensors in earth observation field, the raising of massive multisource remote sensing data acquisition capability, specially
The integrated application of the data acquisition and multi-source data of industry shows the fining division of labor and cooperating type integrated treatment demand and deposits
Develop situation.On the one hand, all kinds of remotely-sensed datas obtain more specialized with way to manage, thus form multiple and different types and defend
Star, different zones or the data center of country；On the other hand, the large-scale synthesis application of remote sensing fields needs to obtain difference again and defends
Star, different zones or the data of National Data Centre are supported, the place for needing while facing dissimilar sensor data is faced with
The realistic problems such as data type, comprehensive information processing complementary with overlay area between reason, different data center.
On the one hand it is total to have expedited the emergence of each data center complex offer data for this series of realistic situations, demand and development trend
The power and technical solution enjoyed, on the other hand, each center complex get up, and provide comprehensive information processing jointly for user
With one-stop information service, become following one of the development trend.For this purpose, constructing more satellite data center collaboration processing and a station
Formula Information Service Mode relies on the satellite datas centers such as domestic and international existing meteorology, land, ocean, studies in more satellite numbers
According under the framework of center handle multi- source Remote Sensing Data data product needed for several key technologies, establish one can unify, cooperative scheduling
The multi- source Remote Sensing Data data collaboration processing platform of more satellite data center resources becomes a pressing issues of remote sensing fields instantly.
Summary of the invention
The object of the present invention is to provide a kind of, and more satellite data central task streams based on nearly data calculating principle dispatch calculation
Method, the goal constraint model calculated by establishing nearly data determine that the workflow execution generalized time minimized determines nearly data
The specific implementation of calculating.Task state is divided per the group by the work of more satellite data center calculation platforms by Hypergraph Theory
It is optimized as stream scheduling method, the transmission time of calculating task its input data in same packets can be made most short.Meanwhile
In the biggish situation of input data amount, substituted using the algorithm Autonomic Migration Framework method between more satellite data center calculation platforms
Large-scale Data Migration.It so can avoid the network interruption that can be encountered during prolonged mass data transfers, storage sky
Between consume the problems such as excessive, thus improve more satellite data centers workflow schedule efficiency and collaboration processing capacity, effectively
Overcome above-mentioned deficiency in the prior art.
The purpose of the present invention is be achieved through the following technical solutions:
A kind of more satellite data central task stream dispatching algorithms based on nearly data calculating principle, comprising the following steps:
Step 1: by preconfigured workflow execution generalized time be reduced to data needed for workflow transmission time and
Input data copies the actual treatment time after computing resource to, using time cost as constraint condition, calculates the workflow and holds
The minimum value of row generalized time；
Step 2: according to data resource to be scheduled and computing resource information is obtained, in preconfigured set dispatching principle
Guidance under, select the resource to match to be combined, obtain Virtual Workflow；
Step 3: the task state PGH that is divided per the group based on Hypergraph Theory carries out the Virtual Workflow scheduling in step 2
Optimization；
Step 4: the algorithm between the nearly data computation model of pre-set more satellite data center calculations being carried out automatic
Migration, parsing including algorithm running environment and packaging method analysis and algorithm compile automatically across computation model.
Further, in step 3, the method for optimization include the workflow with identical input data is divided into it is identical
Task groups, the input data copied is reused in same task groups.
Further, in step 4, the parsing of the algorithm running environment is analyzed with packaging method includes:
Step 4-1-1: inquiring the information in preconfigured algorithms library about the algorithm resource, whether determines algorithm resource
Have the path of source code and source code, while the source code under the path is checked with dynamic link library file；
Step 4-1-2: the environmental variance of algorithm executable file operation user is parsed, when extracting algorithm operation
Environmental variance title needed for dynamic link library, and save as xml file format；
Step 4-1-3: running relied on dynamic link library to algorithm and be packaged, and make the encapsulation of dynamic link library with
The compression method of preconfigured Algorithm source code is consistent.
Further, in step 4, compiling automatically across computation model for the algorithm includes:
Step 4-2-1: the compressed file format being pre-configured in system platform with source code and dynamic link library is utilized
The decompression order to match unzips it source code and dynamic link library, and the road after dynamic link library is decompressed
Diameter is stored in preconfigured temporary file；
Step 4-2-2: after algorithm resource migration to target data center, while the dynamic link library that also migration comes
Path corresponding to file is added in environmental variance, and dynamic link library file path recorded in step 4-2-1 is added more
Newly into the environmental variance of active user, and judge in configuration file whether existing environmental variance of the same name, there is no of the same name
In the case where environmental variance, environmental variance is created and to its assignment；
Step 4-2-3: being called preconfigured automatic compilation script, is called by pre-set ssh agreement
The preconfigured algorithm at target data center compiles Make File file, and driving algorithm resource compilation process executes automatically；
Step 4-2-4: the algorithm resource after compiling successfully generates the executable file to match, passes through preconfigured number
Increase the new algorithm record in target data CENTER ALGORITHM library according to library operation interface.
Further, in step 4-2-2, there are environmental variance of the same name, which is chased after
Add.
The invention has the benefit that nearly data calculating principle is real in more satellite data central task streams are scheduled
The purpose close to data is now calculated, workflow is farthest reduced and always executes the time, to greatly improve more satellite datas
The workflow schedule efficiency at center, in addition, being directed to extensive involved in the multicenter collaboration treatment process of mass remote sensing data
Data migration problems, under more satellite data center environments nearly datamation stream dispatching algorithm can be migrated with algorithm replace it is extensive
It is excessive etc. to can avoid the network interruption that can be encountered during prolonged mass data transfers, memory space consumption for Data Migration
Problem, can one-stop remotely-sensed data service of goods mode of the rapid build based on more satellite data centers framework, and for reply
The complicated demand of multi-user provides technical support, is conducive to the popularization and application in market.
Detailed description of the invention
It in order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, below will be to institute in embodiment
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention
Example, for those of ordinary skill in the art, without creative efforts, can also obtain according to these attached drawings
Obtain other attached drawings.
Fig. 1 is a kind of more satellite data central task based on nearly data calculating principle described according to embodiments of the present invention
Flow the schematic diagram of dispatching algorithm；
Fig. 2 is described based on computing resource processing capacity and critical data resource transmission cost according to embodiments of the present invention
Virtual Workflow dynamic construction schematic diagram；
Fig. 3 is the schematic diagram that task state is divided per the group described in the embodiment of the present invention based on Hypergraph Theory；
Fig. 4 is the process of the algorithm Autonomic Migration Framework between more satellite data center calculation platforms described in the embodiment of the present invention
Figure.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art's every other embodiment obtained belong to what the present invention protected
Range.
As shown in Figs 1-4, a kind of more satellite datas based on nearly data calculating principle according to embodiments of the present invention
Central task stream dispatching algorithm, comprising the following steps:
Step 1: construction work stream executes the nearly data computation model that generalized time minimizes, and the workflow execution is comprehensive
The conjunction time is reduced to the transmission time of data needed for workflow and input data copies the actual treatment time after computing resource to
Two parts, the minimum value of the workflow execution generalized time is calculated using time cost as constraint condition, and what is as minimized is close
Data computation model；
Step 2: the Virtual Workflow dynamic construction based on computing resource processing capacity Yu critical data resource transmission cost
Method, before constructing Virtual Workflow, according to data resource to be scheduled and computing resource information is obtained, preconfigured
Under the guidance of set dispatching principle, selects the resource to match to be combined, obtain Virtual Workflow；
Step 3: the task state PGH that is divided per the group based on Hypergraph Theory carries out the Virtual Workflow scheduling in step 2
Optimization；
The method of optimization includes that the workflow with identical input data is divided into identical task groups, in same task
The input data that recycling had copied in group.
Step 4: the algorithm between the nearly data computation model of pre-set more satellite data center calculations being carried out automatic
Migration, parsing including algorithm running environment and packaging method analysis and algorithm compile automatically across computation model.
Wherein, in step 4, the parsing of the algorithm running environment is analyzed with packaging method includes:
Step 4-1-1: the letter in preconfigured algorithms library about the algorithm resource is inquired in algorithm resource completeness inspection
Breath determines whether algorithm resource has a path of source code and source code, at the same under the path source code and dynamic link
Library file is checked；
Step 4-1-2: algorithm resource environment variable resolution carries out the environmental variance of algorithm executable file operation user
Parsing, environmental variance title needed for dynamic link library when extracting algorithm operation, and save as xml file format；
Step 4-1-3: running relied on dynamic link library to algorithm and be packaged, encapsulation to dynamic link library and pre-
The compression method of the Algorithm source code first configured is consistent.
Wherein, in step 4, compiling automatically across computation model for the algorithm includes:
Step 4-2-1: the decompression of source code and dynamic link library file, using being pre-configured in system platform and source
The decompression order that the compressed file format of code and dynamic link library matches decompresses source code and dynamic link library
Contracting, and the path after dynamic link library is decompressed is stored in preconfigured temporary file；
Step 4-2-2: environmental variance updates, and after algorithm resource migration to target data center, while will also migrate
Path corresponding to the dynamic link library file come is added in environmental variance, by dynamic link recorded in step 4-2-1
Library file path adds update into the environmental variance of active user, and judges whether existing environment of the same name becomes in configuration file
Amount,
In the case where environmental variance of the same name is not present, environmental variance is created and to its assignment；
There are environmental variance of the same name, which is added.
Step 4-2-3: being called preconfigured automatic compilation script, is called by pre-set ssh agreement
The preconfigured algorithm at target data center compiles MakeFile file, and driving algorithm resource compilation process executes automatically；
Step 4-2-4: algorithm resource registering, the algorithm resource after compiling successfully generate the executable file to match, lead to
The new algorithm that preconfigured database manipulation interface increases in target data CENTER ALGORITHM library is crossed to record.
For the nearly data computation model that workflow execution generalized time minimizes, nearly data computation model emphasis considers work
Make the selection in stream input data source, two principal elements of selection of satellite data center calculation resource node, workflow always executes
Time rule is that transmission time, the input data of data needed for workflow copy the actual treatment time two after computing resource to
Point, in order to make workflow always execute time minimum, nearly data model provides the constraint condition of realization process for data resource biography
The defeated time is most short and the computing resource processing time is most short.
Wherein, for the virtual work flowable state structure based on computing resource processing capacity Yu critical data resource transmission cost
Construction method carries out resource allocation under more satellite data central platforms, the simulation process of scheduling need to construct Virtual Workflow, virtual work
It is mainly to consider element that the building for making to flow, which is with the computing capability of computing resource and transmitting data resources cost,.Nearly data calculate
Principle is using data transmission cost as the primary factor of scheduling, then difference between comparing calculation resource, so that dynamic construction is virtual
Workflow.
Wherein, can be related under the framework of more satellite data centers, in the production process of multi- source Remote Sensing Data data product multiple
When the collaboration processing at satellite data center, the time of the repetition copy of shared input data will will increase dramatically workflow schedule mistake
Time cost in journey.The method of the present invention can use PGH as a result, and packet task model is introduced into more satellite data centre frames
In multi- source Remote Sensing Data data process of producing product under structure, the dispatching method of multiple workflows of shared input data is carried out excellent
Change.
For the work of more satellite data center calculation platforms based on PGH (Hypergraph Theory task state is divided per the group)
Make stream method for optimizing scheduling, at the collaboration at the multiple satellite data centers being related in the production process of multi- source Remote Sensing Data data product
When reason, each data center is providing multiple data for the reuse of other computing resources, and the time of data copy will be substantially
Increase the time cost of workflow schedule.The division mode that more shared input files are utilized with PGH, will have identical input data
Workflow be divided into identical task groups, the input data copied is reused in same task groups.
In addition, between more satellite data center calculation platforms in the biggish situation of input data amount, based on nearly data
The algorithm Autonomic Migration Framework method of calculation is migrated instead of large-scale data.The method of the present invention can assist at more satellite data centers as a result,
It can be avoided the network interruption that can be encountered during prolonged mass data transfers in the process with processing, memory space consumed
The problems such as big.
For the algorithm Autonomic Migration Framework method between more satellite data center calculation platforms, nearly data calculate another method
It can be realized by algorithm resource Autonomic Migration Framework, i.e., the distributed migration of algorithm resource replaces extensive between more satellite data centers
Data Migration.Firstly, being analyzed in the algorithm running environment parsing carried out in former data center where algorithm with packaging method.Then
It needs to be compiled automatically in the data center moved in algorithm, the calculating that algorithm information is registered to target data center is put down
On platform.
When concrete application, 1) the nearly data computation model that workflow execution generalized time minimizes: total execution of workflow
Time is reduced to the transmission time of data needed for workflow, input data copies the actual treatment time two after computing resource to
Point；Wherein, nearly data computation model is also referred to as computing platform.
Wherein,It is that the data that workflow WF needs to transmit when being executed data resource to computing resource pass
The defeated time；It is after completing data transmission, in computing resourceTime consumed by upper execution workflow.
It is exactly to ask by the nearly data computation model of constraint condition of time costMinimum value,
I.e.
Wherein, data resourceTo computing resourceTransmission time need to consider network bandwidth between the two, if two
Person is located at the same data center, then the fractional transmission time can be ignored.And under normal conditions, calculating need to consider the two
Between network condition, that is, pass throughTo calculate data transmission period.
Wherein,The calculating for being needs the attribute set by means of computing resource, i.e.,
Wherein (Node-name is the title of computing resource, and CPU-speed is the CPU dominant frequency of computing resource, CPU-
Usage is the CPU current time utilization rate of computing resource, and Memory-size is the memory size of computing resource, Memory-
Usage is the current time utilization rate of computing resource, and Disk-capacity is the free memory of computing resource.
Every impact factor in the set can assign different weighted values respectively according to actual needs.With computing resource phase
The performance data of pass can be updated according to lesser time interval, but also needed to calculate each data resource and calculated and provide
Then the time cost and available network bandwidth transmitted between source provides the available computing resource column of each back end
Table, and be ranked up according to time cost consumed by unit of transfer's data therebetween.
2) the Virtual Workflow dynamic fixing method based on computing resource processing capacity Yu critical data resource transmission cost:
In the Virtual Workflow employed herein based on computing resource processing capacity Yu critical data resource transmission cost
Dynamic fixing method is as shown in Fig. 2, it is using the computing capability of computing resource and transmitting data resources cost as virtual work
Stream chooses the evaluation points of resource, since the basic premise of scheduling is nearly data calculating, therefore using data transmission cost as scheduling
The primary factor, first judge the size of data volume, then difference between comparing calculation resource again.Two data are compared in the figure
The data transmission cost of source Data.a and Data.b, judgement schematics are as follows:
Wherein,It is the input data of workflow,、It is data position and data volume two respectively
The weight factor of index.
The building of Virtual Workflow is main consideration element with the computing capability of computing resource and transmitting data resources cost
, wherein computing resource ability is primarily referred to as after the type and data scale for determining pending data resource, with data resource
Centered on judge whether the algorithm resource at center where data matches with the data source, under the premise of the two is mutually matched, then
Pair it is possible that transmitting data resources situation carry out transmission cost calculating.
3) the workflow schedule optimization method of more satellite data center calculation platforms based on PGH
Use herein based on Hypergraph Theory be divided per the group task state as shown in figure 3, in computing resource and data
In situation known to network bandwidth between resource, expected execute time (EET) of workflow is the data with input file
It measures linearly increasing, in fact, the calculation method due to EET is extremely complex, but herein to simplifies the calculating of time cost
Process, the calculating of EET only consider from data center to when the execution of the time of computing unit copy data and each processing task
Between this two parts time cost.
In formula,It is input fileSize,It is transmission fileWhen each byte paid
Cost,It is the processing cost of each byte in file f j.
Wherein,It can be calculate by the following formula:
It is processing workflowFirst probability being performed in being organized where it,It is
Required input file is provided simultaneously on the computing unit that it is executedProbability, BW is data center where data resource
With the network bandwidth between data center where computing unit.
According to the regulation of above formula, the process flow with identical input data will be divided into one group, so that EET () reach
To minimum, aforementioned packet task model can be divided into tri- task groups of P1, P2, P3,
4) the algorithm Autonomic Migration Framework method between more satellite data center calculation platforms:
The algorithm Autonomic Migration Framework method between more satellite data center calculation platforms used herein is as shown in figure 4, main
Parsing and packaging method analysis and algorithm including algorithm running environment compile two steps across computing platform automatically.
Wherein, the step of parsing of algorithm running environment and packaging method are analyzed include:
1, algorithm resource completeness inspection: the information first in query operator Faku County about the algorithm resource determines that algorithm provides
Whether source has the path of source code, source code, while is checked with dynamic link library file the source code under the path.
2, algorithm resource environment variable resolution: algorithm resource is also at runtime to require to environmental variance, is led to herein
It crosses and the environmental variance of algorithm executable file operation user is parsed, when extracting algorithm operation needed for dynamic link library
Environmental variance title, and save as xml file format.
3, algorithm runs the encapsulation of relied on dynamic link library, the encapsulation of dynamic link library and the compression of Algorithm source code
Method is consistent, is all the compressing file tool by means of Linux system platform, is packaged into the compression text that suffix is * .tar.gz
Part reduces volume of transmitted data on the basis of guaranteeing transmission security of file.
By environmental variance parse with save, the compression of Algorithm source code and encapsulation, the relied on dynamic link library of algorithm
Compression is with after encapsulation, and preparation of the algorithm migration in data center local is just completed, and the output file in stage includes being used for
The xml document of storage environment variable, two tar packet compressed files for encapsulating Algorithm source code and dynamic link library file.
In addition, algorithm includes: across the method and step that computing platform compiles automatically
1, the decompression of source code and dynamic link library file；The compressed file format of source code and dynamic link library is
.tar.gz, the decompression order for the Linux system platform that can be called directly is decompressed to it；And after dynamic link library decompression
Path also need to be stored temporarily in temporary file because the path and algorithm after decompression where dynamic link library are in former number
Can not be completely the same according to supercentral path, so needing to keep in the dynamic link library path of all decompressions, in order to update
Environmental variance.
2, environmental variance updates: after algorithm resource migration to target data center, while the dynamic that also migration comes
Path corresponding to link library file is added in environmental variance, can be by dynamic link library file path recorded in upper step
Add update into the environmental variance of active user, if having environmental variance of the same name in configuration file, the value is added, if not
There are environmental variances, then can create and to its assignment.
3, the calling of automatic compilation script: the execution of automatic compilation script is by ssh agreement invocation target data
The algorithm of the heart compiles Make File file, drives the automatic execution of algorithm resource compilation process.
4, algorithm resource registering: the algorithm resource after compiling successfully can generate executable file, in order to record the algorithm
The information such as title, path, parameter, the new algorithm that can be increased in target data CENTER ALGORITHM library by database manipulation interface are remembered
Record.
According to described above, design completes a kind of more satellite data central task stream tune based on nearly data calculating principle
Algorithm is spent, main task is to realize to calculate the purpose close to data in the workflow schedule of more satellite data centers framework,
To which the total run time of workflow is minimized.This method utilizes nearly data calculating principle, in more satellite data centre frames
It in the workflow schedule of structure, realizes and calculates the purpose close to data, farthest reduce workflow and always execute the time.
In conclusion this programme proposes a kind of former based on the calculating of nearly data by means of above-mentioned technical proposal of the invention
More satellite data central task stream dispatching algorithms then, the nearly data computation model that workflow execution generalized time is minimized are made
For cost constraint model, and utilize the dynamic construction based on computing resource processing capacity Yu critical data resource transmission cost method
Virtual Workflow, to carry out resource allocation and scheduling under more satellite data central platforms；In the excellent of workflow schedule method
In change, shared data is reused using the task state PGH that is divided per the group of Hypergraph Theory, is reduced between each data center
Data Migration amount.In addition, the distributed migration in algorithm resource replaces the large-scale data between more satellite data centers to move
Move on realizing that nearly data calculate, using the parsing of algorithm running environment and packaging method analysis and algorithm across computing platform certainly
Dynamic Compilation Method can complete the algorithm Autonomic Migration Framework towards more cluster environment.
Therefore, the present invention can efficiently dispatch each satellite number in actual more satellite data central task stream scheduling
It, can be according to the characteristics of multi- source Remote Sensing Data data product treatment process and knot according to the multi- source Remote Sensing Data data collection and computing resource at center
It closes mass data distributed storage status and transmission consumes, efficient process flow scheduling is carried out, to promote in satellite data
The collaboration of the heart is handled；In addition, the multicenter for mass remote sensing data cooperates with the migration of large-scale data involved in treatment process
Problem, nearly datamation stream dispatching algorithm can be migrated with algorithm under more satellite data center environments move instead of large-scale data
It moves, can avoid the network interruption that can be encountered during prolonged mass data transfers, memory space consumes the problems such as excessive, energy
Enough one-stop remotely-sensed data service of goods mode of the rapid build based on more satellite data centers framework, and be reply multi-user's
Complicated demand provides technical support, is conducive to the popularization and application in market.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (4)
1. a kind of more satellite data central task stream dispatching algorithms based on nearly data calculating principle, which is characterized in that including with
Lower step:
Step 1: preconfigured workflow execution generalized time is reduced to the transmission time and input of data needed for workflow
The actual treatment time after data copy to computing resource, it is comprehensive to calculate the workflow execution using time cost as constraint condition
Close the minimum value of time；
Step 2: according to data resource to be scheduled and computing resource information is obtained, in the finger of preconfigured set dispatching principle
It leads down, selects the resource to match to be combined, obtain Virtual Workflow；
Step 3: the task state PGH that is divided per the group based on Hypergraph Theory optimizes the Virtual Workflow scheduling in step 2,
The method of optimization includes that the workflow with identical input data is divided into identical task groups, is repeated in same task groups
Utilize the input data copied；
Step 4: Autonomic Migration Framework is carried out to the algorithm between the nearly data computation model in pre-set more satellite data centers, including
The parsing of algorithm running environment and packaging method are analyzed and algorithm compiles automatically across computation model, wherein nearly data calculate mould
Type is also referred to as computing platform.
2. more satellite data central task stream dispatching algorithms according to claim 1 based on nearly data calculating principle,
It is characterized in that, in step 4, parsing and the packaging method analysis of the algorithm running environment include:
Step 4-1-1: inquiring the information in preconfigured algorithms library about the algorithm resource, determines whether algorithm resource has
The path of source code and source code, while the source code under the path is checked with dynamic link library file；
Step 4-1-2: the environmental variance of algorithm executable file operation user is parsed, dynamic when extracting algorithm operation
Environmental variance title needed for chained library, and save as xml document format；
Step 4-1-3: running relied on dynamic link library to algorithm and be packaged, and makes the encapsulation and in advance of dynamic link library
The compression method of the Algorithm source code of configuration is consistent.
3. more satellite data central task stream dispatching algorithms according to claim 1 based on nearly data calculating principle,
It is characterized in that, in step 4, compiling automatically across computation model for the algorithm includes:
Step 4-2-1: the compressed file format phase being pre-configured in system platform with source code and dynamic link library is utilized
The decompression order matched unzips it source code and dynamic link library, and the path after dynamic link library is decompressed is protected
There are in preconfigured temporary file；
Step 4-2-2: after algorithm resource migration to target data center, while the dynamic link library file that also migration comes
Corresponding path is added in environmental variance, by dynamic link library file routing update recorded in step 4-2-1 to working as
In the environmental variance of preceding user, and judge in configuration file whether existing environmental variance of the same name, there is no environment of the same name to become
In the case where amount, environmental variance is created and to its assignment；
Step 4-2-3: being called preconfigured automatic compilation script, passes through pre-set ssh agreement invocation target
The preconfigured algorithm of data center compiles Make File file, and driving algorithm resource compilation process executes automatically；
Step 4-2-4: the algorithm resource after compiling successfully generates the executable file to match, passes through preconfigured database
Operation interface increases the new algorithm record in target data CENTER ALGORITHM library.
4. more satellite data central task stream dispatching algorithms according to claim 3 based on nearly data calculating principle,
It is characterized in that, in step 4-2-2, there are environmental variance of the same name, which is added.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410851865.9A CN104484230B (en) | 2014-12-31 | 2014-12-31 | More satellite data central task stream dispatching algorithms based on nearly data calculating principle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410851865.9A CN104484230B (en) | 2014-12-31 | 2014-12-31 | More satellite data central task stream dispatching algorithms based on nearly data calculating principle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104484230A CN104484230A (en) | 2015-04-01 |
| CN104484230B true CN104484230B (en) | 2019-03-15 |
Family
ID=52758774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410851865.9A Active CN104484230B (en) | 2014-12-31 | 2014-12-31 | More satellite data central task stream dispatching algorithms based on nearly data calculating principle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104484230B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656799B (en) * | 2017-11-06 | 2021-06-25 | 福建师范大学 | Workflow scheduling method considering communication and computing cost under multi-cloud environment |
| CN108985709A (en) * | 2018-06-26 | 2018-12-11 | 中国科学院遥感与数字地球研究所 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
| CN111475301B (en) * | 2020-04-09 | 2021-06-11 | 清华大学 | Satellite resource allocation method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315424A (en) * | 2008-07-29 | 2008-12-03 | 中国科学院对地观测与数字地球科学中心 | Multi-satellite remote sensing data integrated parallel ground pretreatment system |
| CN102289540A (en) * | 2011-07-01 | 2011-12-21 | 中国航空工业集团公司科学技术委员会 | Workflow-driven genetic algorithm aviation optimization system orienting to service heterogeneous grid |
| CN104239143A (en) * | 2014-09-17 | 2014-12-24 | 中国科学院遥感与数字地球研究所 | Satellite data receiving task scheduling system based on OSGI (Open Service Gateway Initiative) and workflow and satellite data receiving task scheduling method based on OSGI and workflow |
-
2014
- 2014-12-31 CN CN201410851865.9A patent/CN104484230B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315424A (en) * | 2008-07-29 | 2008-12-03 | 中国科学院对地观测与数字地球科学中心 | Multi-satellite remote sensing data integrated parallel ground pretreatment system |
| CN102289540A (en) * | 2011-07-01 | 2011-12-21 | 中国航空工业集团公司科学技术委员会 | Workflow-driven genetic algorithm aviation optimization system orienting to service heterogeneous grid |
| CN104239143A (en) * | 2014-09-17 | 2014-12-24 | 中国科学院遥感与数字地球研究所 | Satellite data receiving task scheduling system based on OSGI (Open Service Gateway Initiative) and workflow and satellite data receiving task scheduling method based on OSGI and workflow |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104484230A (en) | 2015-04-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104898573B (en) | A kind of digital control system data acquisition based on cloud computing and processing method | |
| CN103019742B (en) | A kind of real time workshop method in many DSP platform | |
| CN107423823B (en) | R language-based machine learning modeling platform architecture design method | |
| JP2014525640A (en) | Expansion of parallel processing development environment | |
| CN109583002B (en) | Avionics integrated design method | |
| CN104598425A (en) | General multiprocessor parallel calculation method and system | |
| CN105843182A (en) | Power dispatching accident handling scheme preparing system and power dispatching accident handling scheme preparing method based on OMS | |
| CN102841816A (en) | Method for parallel processing of remote sensing imagery data | |
| CN104915482A (en) | Satellite data receiving simulation analysis platform | |
| Qian et al. | An energy-saving task scheduler for mobile devices | |
| Song et al. | A recursive operator allocation approach for assembly line-balancing optimization problem with the consideration of operator efficiency | |
| CN105094984A (en) | Resource scheduling method and system | |
| CN104484230B (en) | More satellite data central task stream dispatching algorithms based on nearly data calculating principle | |
| CN105573760A (en) | Storm-based processing system and method of data of Internet of Things | |
| CN102375734A (en) | Application product development system, method and device and operation system, method and device | |
| CN112130812B (en) | Analysis model construction method and system based on data stream mixed arrangement | |
| CN100478883C (en) | Plug-in component, system and method for providing unionize interface for multicell application | |
| CN107203421A (en) | A kind of adaptive work in combination stream method in cloud computing environment | |
| Aksakalli et al. | Systematic approach for generation of feasible deployment alternatives for microservices | |
| CN102780583A (en) | Method for service description, service combination and service quality assessment of Internet of Things | |
| CN115756833A (en) | AI inference task scheduling method and system oriented to multiple heterogeneous environments | |
| Pajic et al. | Robust architectures for embedded wireless network control and actuation | |
| CN102103519A (en) | Method for optimizing Web service execution based on scripts | |
| Chen et al. | AndroidOff: Offloading android application based on cost estimation | |
| CN101872302B (en) | BPEL-based modeling method for control information flow graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211103 Address after: No. 19, North Fourth Ring West Road, Haidian District, Beijing 100873 Patentee after: Aerospace Information Research Institute,Chinese Academy of Sciences Address before: No. 9 Dengzhuang South Road, Haidian District, Beijing 100094 Patentee before: Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences |
|
| TR01 | Transfer of patent right |