CN104484230A - Multiple satellite data centre workflow scheduling algorithm on basis of near data calculation principle - Google Patents
Multiple satellite data centre workflow scheduling algorithm on basis of near data calculation principle Download PDFInfo
- Publication number
- CN104484230A CN104484230A CN201410851865.9A CN201410851865A CN104484230A CN 104484230 A CN104484230 A CN 104484230A CN 201410851865 A CN201410851865 A CN 201410851865A CN 104484230 A CN104484230 A CN 104484230A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- data
- workflow
- resource
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a multiple satellite data centre workflow scheduling algorithm on the basis of a near data calculation principle. The algorithm comprises the following steps of (1) constructing a near data calculation model with the minimum workflow execution combined time; (2) dynamically constructing a virtual workflow based on a computing resource processing capacity and a key data resource transmission cost; (3) optimizing virtual workflow scheduling in the step (2) based on the grouped task partitioning way of a super-graph theory (PGH); (4) automatically migrating preset algorithms for the calculation of near data calculation models between multiple satellite data centres. The algorithm has the beneficial effects of furthest shortening the workflow total execution time so as to improve the workflow scheduling efficiency of the multiple satellite data centres, fast constructing a one-stop remote sensing data product service mode based on the architecture of the multiple satellite data centres, and providing technical support for the complicated demands of multiple users.
Description
Technical field
The present invention relates to a kind of towards many satellite datas central task stream dispatching algorithm, specifically, relate to a kind of many satellite datas central task stream dispatching algorithm based on nearly data calculating principle.
Background technology
Along with the development of multiple sensors in earth observation field, the raising of magnanimity RS data acquisition capability, the integrated application of specialized data acquisition and multi-source data presents to become more meticulous divides the work with cooperating type overall treatment demand and the development situation of depositing.On the one hand, all kinds of remotely-sensed data obtains with way to manage more specialized, thus form the data center of multiple dissimilar satellite, zones of different or country; On the other hand, the large-scale synthesis application of remote sensing fields needs again to obtain the Data support of different satellite, zones of different or National Data Centre, is faced with and needs simultaneously in the face of the realistic problem such as complementation, comprehensive information process of data type and overlay area between the process, different pieces of information center of dissimilar sensor data.
This series of realistic situation, Requirement and development trend, expedite the emergence of power and the technical solution that each data center complex provides data sharing on the one hand, on the other hand, each center complex gets up, jointly for user provides comprehensive information processing and one-stop information service, become one of the development trend in future.For this reason, build many satellite datas center associated treatment and one-stop Information Service Mode, rely on the satellite data centers such as domestic and international existing meteorology, land, ocean, study some gordian techniquies of processing under the framework of many satellite datas center needed for RS data product, set up that one can be unified, the RS data associated treatment platform of cooperative scheduling many satellite datas center resources becomes pressing issues of remote sensing fields instantly.
Summary of the invention
The object of this invention is to provide a kind of many satellite datas central task stream dispatching algorithm based on nearly data calculating principle, by setting up the goal constraint model that nearly data calculate, determine that minimized workflow execution generalized time determines the specific implementation that nearly data calculate.The workflow schedule method of many satellite datas center calculation platform being optimized by group division task state by Hypergraph Theory, can make transmission time of its input data of the calculation task in same packets the shortest.Meanwhile, when inputting data volume and being larger, the algorithm Autonomic Migration Framework method between many satellite datas center calculation platform is utilized to substitute large-scale Data Migration.The network interruption so can avoiding running in long mass data transfers process, storage space consume the problems such as excessive, thus improve workflow schedule efficiency and the associated treatment ability at many satellite datas center, effectively overcome above-mentioned deficiency of the prior art.
The object of the invention is to be achieved through the following technical solutions:
Based on many satellite datas central task stream dispatching algorithm of nearly data calculating principle, comprise the following steps:
Step 1: pre-configured workflow execution generalized time is reduced to the transmission time of workflow desired data and the actual treatment time after inputting data copy to computational resource, be constraint condition with time cost, calculate the minimum value of described workflow execution generalized time;
Step 2: according to obtaining data resource to be scheduled and computational resource information, under the guidance of pre-configured set dispatching principle, selects the resource matched to combine, obtains Virtual Workflow;
Step 3: the Virtual Workflow scheduling in step 2 is optimized by group division task state PGH based on Hypergraph Theory;
Step 4: Autonomic Migration Framework is carried out to the algorithm between the nearly data computation model of the many satellite datas center calculation pre-set, comprise the parsing of algorithm running environment and method for packing analysis and algorithm across computation model automatic compiling.
Further, in step 3, the method for optimization comprises the workflow with identical input data is divided into identical task groups, reuses the input data copied in same task groups.
Further, in step 4, the parsing of described algorithm running environment and method for packing analysis comprise:
Step 4-1-1: inquire about the information about this algorithm resource in pre-configured algorithms library, determines whether algorithm resource possesses the path of source code and source code, checks simultaneously to the source code under this path and dynamic link library file;
Step 4-1-2: resolve the environmental variance of algorithm executable file run user, extracts the environmental variance title needed for dynamic link library when algorithm runs, and saves as xml file layout;
Step 4-1-3: the dynamic link library relied on is run to algorithm and encapsulates, and make the encapsulation of dynamic link library consistent with the compression method of pre-configured Algorithm source code.
Further, in step 4, the comprising across computation model automatic compiling of described algorithm:
Step 4-2-1: utilize and be pre-configured in the decompression order matched with the compressed file format of source code and dynamic link library in system platform source code and dynamic link library are decompressed, and the path after dynamic link library decompression is kept in pre-configured temporary file;
Step 4-2-2: after algorithm resource migration to target data center, the path corresponding to dynamic link library file simultaneously also migration will come is added in environmental variance, being added in the dynamic link library file path of recording in step 4-2-1 is updated in the environmental variance of active user, and judge whether there is environmental variance of the same name in configuration file, when there is not environmental variance of the same name, newly-built environmental variance to its assignment;
Step 4-2-3: call pre-configured automatic compiling script, compiles Make File file by the pre-configured algorithm of the ssh agreement invocation target data center pre-set, and drives algorithm resource compilation process to automatically perform;
Step 4-2-4: compile the algorithm resource successfully and generate the executable file matched, increases the new algorithm record in target data CENTER ALGORITHM storehouse by pre-configured database manipulation interface.
Further, in step 4-2-2, when there is environmental variance of the same name, this environmental variance is added.
Beneficial effect of the present invention is: in many satellite datas central task stream is dispatched, nearly data calculating principle realizes calculating to the close object of data, farthest reduce workflow total execution time, thus greatly improve the workflow schedule efficiency at many satellite datas center, in addition, for the large-scale data migration problem related in the multicenter associated treatment process of mass remote sensing data, under many satellite datas center environment, nearly datamation stream dispatching algorithm can replace large-scale data migration with algorithm migration, the network interruption that can run in long mass data transfers process can be avoided, storage space consumes the problems such as excessive, can rapid build based on the one-stop remotely-sensed data service of goods pattern of many satellite datas center framework, and provide technical support for tackling the complicated demand of multi-user, be conducive to the propagation and employment in market.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the schematic diagram of a kind of many satellite datas central task stream dispatching algorithm based on nearly data calculating principle according to the embodiment of the present invention;
Fig. 2 is the Virtual Workflow dynamic construction schematic diagram based on computational resource processing power and critical data resource transmission cost according to the embodiment of the present invention;
Fig. 3 is the schematic diagram dividing task state by group based on Hypergraph Theory described in the embodiment of the present invention;
Fig. 4 is the process flow diagram of the algorithm Autonomic Migration Framework between the many satellite datas center calculation platform described in the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain, all belongs to the scope of protection of the invention.
As Figure 1-4, a kind of many satellite datas central task stream dispatching algorithm based on nearly data calculating principle according to the embodiment of the present invention, comprises the following steps:
Step 1: construction work stream performs the minimized nearly data computation model of generalized time, described workflow execution generalized time is reduced to the transmission time of workflow desired data and the actual treatment time two parts inputted after data copy to computational resource, take time cost as the minimum value that constraint condition calculates described workflow execution generalized time, be minimized nearly data computation model;
Step 2: based on the Virtual Workflow dynamic fixing method of computational resource processing power and critical data resource transmission cost, before structure Virtual Workflow, according to obtaining data resource to be scheduled and computational resource information, under the guidance of pre-configured set dispatching principle, select the resource matched to combine, obtain Virtual Workflow;
Step 3: the Virtual Workflow scheduling in step 2 is optimized by group division task state PGH based on Hypergraph Theory;
The method optimized comprises the workflow with identical input data is divided into identical task groups, reuses the input data copied in same task groups.
Step 4: Autonomic Migration Framework is carried out to the algorithm between the nearly data computation model of the many satellite datas center calculation pre-set, comprise the parsing of algorithm running environment and method for packing analysis and algorithm across computation model automatic compiling.
Wherein, in step 4, the parsing of described algorithm running environment and method for packing analysis comprise:
Step 4-1-1: algorithm resource completeness checks, inquire about the information about this algorithm resource in pre-configured algorithms library, determine whether algorithm resource possesses the path of source code and source code, the source code under this path and dynamic link library file are checked simultaneously;
Step 4-1-2: algorithm resource environment variable resolution, resolves the environmental variance of algorithm executable file run user, extracts the environmental variance title needed for dynamic link library when algorithm runs, and saves as xml file layout;
Step 4-1-3: the dynamic link library relied on is run to algorithm and encapsulates, consistent with the compression method of pre-configured Algorithm source code to the encapsulation of dynamic link library.
Wherein, in step 4, the comprising across computation model automatic compiling of described algorithm:
Step 4-2-1: the decompression of source code and dynamic link library file, utilize and be pre-configured in the decompression order matched with the compressed file format of source code and dynamic link library in system platform source code and dynamic link library are decompressed, and the path after dynamic link library decompression is kept in pre-configured temporary file;
Step 4-2-2: environmental variance upgrades, after algorithm resource migration to target data center, the path corresponding to dynamic link library file simultaneously also migration will come is added in environmental variance, being added in the dynamic link library file path of recording in step 4-2-1 is updated in the environmental variance of active user, and judge whether there is environmental variance of the same name in configuration file
When there is not environmental variance of the same name, newly-built environmental variance to its assignment;
When there is environmental variance of the same name, this environmental variance is added.
Step 4-2-3: call pre-configured automatic compiling script, compiles MakeFile file by the pre-configured algorithm of the ssh agreement invocation target data center pre-set, and drives algorithm resource compilation process to automatically perform;
Step 4-2-4: algorithm resource registering, compiles the algorithm resource successfully and generates the executable file matched, and increases the new algorithm record in target data CENTER ALGORITHM storehouse by pre-configured database manipulation interface.
For the minimized nearly data computation model of workflow execution generalized time, nearly data computation model emphasis considers the workflow input selection of data source, selection two principal elements of satellite data center calculation resource node, workflow total execution time be defined as workflow desired data transmission time, input data copy to computational resource after actual treatment time two parts, in order to make workflow total execution time minimize, nearly data model specifies that the constraint condition of implementation procedure is that transmitting data resources shortest time and computational resource processing time are the shortest.
Wherein, for the Virtual Workflow dynamic fixing method based on computational resource processing power and critical data resource transmission cost, carry out Resourse Distribute under many satellite datas central platform, the simulation process of scheduling need build Virtual Workflow, the structure of Virtual Workflow is mainly consider key element with the computing power of computational resource and transmitting data resources cost.Nearly data calculating principle is using the primary factor of data transmission cost as scheduling, then difference between comparing calculation resource, thus dynamic construction Virtual Workflow.
Wherein, under the framework of many satellite datas center, when can relate to the associated treatment at multiple satellite data center in the production run of RS data product, the time repeating to copy of shared input data significantly will increase the time cost in workflow schedule process.Thus, the inventive method can utilize PGH, is incorporated into by bag task model in the RS data process of producing product under the framework of many satellite datas center, is optimized the dispatching method of multiple workflows of shared input data.
For the workflow schedule optimization method of the many satellite datas center calculation platform based on PGH (the dividing task state by group of Hypergraph Theory), during the associated treatment at the multiple satellite data centers related in the production run of RS data product, each data center reuses for other computational resources providing multiple data, and the time of data copy significantly will increase the time cost of workflow schedule.Share to the dividing mode that input file utilizes PGH more, the workflow with identical input data is divided into identical task groups, in same task groups, reuses the input data copied.
In addition, input between many satellite datas center calculation platform data volume larger, the algorithm Autonomic Migration Framework method calculated based on nearly data replaces large-scale data migration.Thus, the inventive method can be able to be avoided running in long mass data transfers process in the associated treatment process of many satellite datas center network interruption, storage space consume the problems such as excessive.
For the algorithm Autonomic Migration Framework method between many satellite datas center calculation platform, nearly data calculate another kind of method and can be realized by algorithm resource Autonomic Migration Framework, and namely the migration of algorithm distribution of resource formula replaces the large-scale data migration between many satellite datas center.First, the algorithm running environment of the former data center in algorithm place carrying out is resolved and method for packing analysis.Then need at algorithm the data center that moves to carries out automatic compiling, algorithm information is registered in the computing platform at target data center.
During embody rule, 1) the minimized nearly data computation model of workflow execution generalized time: total execution time of workflow is reduced to the actual treatment time two parts after the transmission time of workflow desired data, input data copy to computational resource; Wherein, nearly data computation model is also referred to as computing platform.
Wherein,
that workflow WF needs when performing to transmit the data transmission period of data resource to computational resource;
after data transmission, at computational resource
the time that upper execution workflow consumes.
Be that the nearly data computation model of constraint condition is asked with time cost
minimum value,
Namely
Wherein, data resource
to computational resource
transmission time need to consider the network bandwidth between the two, if the two is positioned at same data center, then can ignore this fractional transmission time.And under normal circumstances, calculating need to consider network condition therebetween, namely pass through
calculate data transmission period.
Wherein,
the calculating being needs the community set by means of computational resource, namely
Wherein (Node-name is the title of computational resource, CPU-speed is the CPU dominant frequency of computational resource, CPU-usage is the CPU current time utilization factor of computational resource, Memory-size is the memory size of computational resource, Memory-usage is the current time utilization factor of computational resource, and Disk-capacity is the free memory of computational resource.
Every factor of influence in this set can give different weighted values according to the actual requirements respectively.The performance data relevant to computational resource can upgrade according to the less time interval, but also need to calculate the time cost and available network bandwidth that to carry out between each data resource and computational resource transmitting, then provide each back end can computational resource list, and to sort according to the time cost that unit of transfer's data consume between.
2) based on the Virtual Workflow dynamic fixing method of computational resource processing power and critical data resource transmission cost:
At the Virtual Workflow dynamic fixing method based on computational resource processing power and critical data resource transmission cost adopted herein as shown in Figure 2, it chooses the evaluation points of resource using the computing power of computational resource and transmitting data resources cost as Virtual Workflow, basic premise due to scheduling is that nearly data calculate, therefore using the primary factor of data transmission cost as scheduling, first judge the size of data volume, and then difference between comparing calculation resource.In the drawings, the data transmission cost of contrast two data source Data.a and Data.b, its judgement schematics is as follows:
Wherein,
the input data of workflow,
,
the weight factor of data position and data volume two indices respectively.
The structure of Virtual Workflow is mainly consider key element with the computing power of computational resource and transmitting data resources cost, wherein, computational resource ability mainly refers to after the type determining pending data resource and data scale, centered by data resource, judge whether the algorithm resource at center, data place mates with this data source, under the prerequisite that the two mates mutually, then the transmitting data resources situation that may occur is carried out to the calculating of transmission cost.
3) based on the workflow schedule optimization method of many satellite datas center calculation platform of PGH
Adopt herein based on Hypergraph Theory by group divide task state as shown in Figure 3, the network bandwidth between computational resource and data resource is known, the expection execution time (EET) of workflow is along with the data volume of input file linearly increases, in fact, because the computing method of EET are very complicated, but herein in order to simplify the computation process of time cost, this two parts time cost of execution time from data center to the time of computing unit copies data and each Processing tasks is only considered in the calculating of EET.
In formula,
it is input file
size,
it is transfer files
time each byte cost of paying,
it is the process cost of each byte in file f j.
Wherein,
calculate by following formula:
work for the treatment of stream
first probability be performed in its place group,
be
the computing unit that it performs possesses required input file simultaneously
probability, BW is the network bandwidth between data resource place data center and computing unit place data center.
According to the regulation of above formula, the treatment scheme with identical input data will be divided into one group, with make EET (
) reach and minimize, aforementioned bag task model can be divided into P1, P2, P3 tri-task groups,
4) the algorithm Autonomic Migration Framework method between many satellite datas center calculation platform:
Algorithm Autonomic Migration Framework method between the many satellite datas center calculation platform adopted herein as shown in Figure 4, mainly comprise the parsing of algorithm running environment and method for packing analysis and algorithm across computing platform automatic compiling two steps.
Wherein, the parsing of algorithm running environment and the step of method for packing analysis comprise:
1, algorithm resource completeness check: first in query operator Faku County about the information of this algorithm resource, determine whether algorithm resource possesses the path of source code, source code, the source code under this path and dynamic link library file checked simultaneously.
2, algorithm resource environment variable resolution: algorithm resource operationally also has requirement to environmental variance, resolve herein by the environmental variance of algorithm executable file run user, extract the environmental variance title needed for dynamic link library when algorithm runs, and save as xml file layout.
3, algorithm runs the encapsulation of the dynamic link library relied on, the encapsulation of dynamic link library is consistent with the compression method of Algorithm source code, it is all the compressing file instrument by means of Linux system platform, be packaged into the compressed file that suffix is * .tar.gz, on the basis ensureing transmission security of file, reduce volume of transmitted data.
Through environmental variance resolve with preserve, compression that the compression of Algorithm source code and encapsulation, algorithm institute rely on dynamic link library and after encapsulating, the preliminary work that algorithm moves heart this locality in the data just completes, the output file in stage comprises, for storage environment variable xml file, encapsulate two tar packet compression files of Algorithm source code and dynamic link library file.
In addition, the method step across computing platform automatic compiling of algorithm comprises:
1, the decompression of source code and dynamic link library file; The compressed file format of source code and dynamic link library is .tar.gz, to the decompression order of the Linux system platform that its decompression can directly be called; And the path after dynamic link library decompression also needs temporarily to be kept in temporary file, because path and the path of algorithm in former data center at the rear dynamic link library place of decompression can not be completely the same, so need the dynamic link library path of temporary all decompress(ion)s, so that upgrade environmental variance.
2, environmental variance upgrades: after algorithm resource migration to target data center, the path corresponding to dynamic link library file simultaneously also migration will come is added in environmental variance, the dynamic link library file path of recording in upper step can be added is updated in the environmental variance of active user, if existing environmental variance of the same name in configuration file, then add this value, if there is no environmental variance, then can be newly-built and to its assignment.
3, the calling of automatic compiling script: the execution of automatic compiling script is the algorithm compiling Make File file by ssh agreement invocation target data center, drives automatically performing of algorithm resource compilation process.
4, algorithm resource registering: compile the algorithm resource successfully and can generate executable file, in order to record the information such as title, path, parameter of this algorithm, increases the new algorithm record in target data CENTER ALGORITHM storehouse by database manipulation interface.
According to above introduction, design completes a kind of many satellite datas central task stream dispatching algorithm based on nearly data calculating principle, main task realizes calculating to the close object of data in the workflow schedule of many satellite datas center framework, thus drop to minimum by the total run time of workflow.This method utilizes nearly data calculating principle, in the workflow schedule of many satellite datas center framework, realizes calculating to the close object of data, farthest reduces workflow total execution time.
In sum, by means of technique scheme of the present invention, this programme proposes a kind of many satellite datas central task stream dispatching algorithm based on nearly data calculating principle, using minimized for workflow execution generalized time nearly data computation model as cost constraint model, and utilize the dynamic construction Virtual Workflow based on computational resource processing power and critical data resource transmission cost method, to carry out Resourse Distribute and scheduling under many satellite datas central platform; In the optimization of workflow schedule method, what have employed Hypergraph Theory will share Data duplication utilization by group division task state PGH, reduce the Data Migration amount between each data center.In addition, replace the migration of the large-scale data between many satellite datas center to realize nearly data in the migration of algorithm distribution of resource formula to calculate, have employed the parsing of algorithm running environment and method for packing analysis and algorithm across computing platform automatic compiling method can finished surface to the algorithm Autonomic Migration Framework of many cluster environment.
Therefore, the present invention is in many satellite datas central task stream scheduling of reality, RS data collection and the computational resource at each satellite data center can be dispatched efficiently, can consume according to the feature of RS data product treatment flow process and in conjunction with mass data distributed storage present situation and transmission, carry out efficient treatment scheme scheduling, thus promote the associated treatment at satellite data center; In addition, for the large-scale data migration problem related in the multicenter associated treatment process of mass remote sensing data, under many satellite datas center environment, nearly datamation stream dispatching algorithm can replace large-scale data migration with algorithm migration, the network interruption can avoiding running in long mass data transfers process, storage space consume the problems such as excessive, can rapid build based on the one-stop remotely-sensed data service of goods pattern of many satellite datas center framework, and provide technical support for tackling the complicated demand of multi-user, be conducive to the propagation and employment in market.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (5)
1., based on many satellite datas central task stream dispatching algorithm of nearly data calculating principle, it is characterized in that, comprise the following steps:
Step 1: pre-configured workflow execution generalized time is reduced to the transmission time of workflow desired data and the actual treatment time after inputting data copy to computational resource, be constraint condition with time cost, calculate the minimum value of described workflow execution generalized time;
Step 2: according to obtaining data resource to be scheduled and computational resource information, under the guidance of pre-configured set dispatching principle, selects the resource matched to combine, obtains Virtual Workflow;
Step 3: the Virtual Workflow scheduling in step 2 is optimized by group division task state PGH based on Hypergraph Theory;
Step 4: Autonomic Migration Framework is carried out to the algorithm between the nearly data computation model of the many satellite datas center calculation pre-set, comprise the parsing of algorithm running environment and method for packing analysis and algorithm across computation model automatic compiling.
2. the many satellite datas central task stream dispatching algorithm based on nearly data calculating principle according to claim 1, it is characterized in that, in step 3, the method optimized comprises the workflow with identical input data is divided into identical task groups, reuses the input data copied in same task groups.
3. the many satellite datas central task stream dispatching algorithm based on nearly data calculating principle according to claim 1, is characterized in that, in step 4, parsing and the method for packing analysis of described algorithm running environment comprise:
Step 4-1-1: inquire about the information about this algorithm resource in pre-configured algorithms library, determines whether algorithm resource possesses the path of source code and source code, checks simultaneously to the source code under this path and dynamic link library file;
Step 4-1-2: resolve the environmental variance of algorithm executable file run user, extracts the environmental variance title needed for dynamic link library when algorithm runs, and saves as xml file layout;
Step 4-1-3: the dynamic link library relied on is run to algorithm and encapsulates, and make the encapsulation of dynamic link library consistent with the compression method of pre-configured Algorithm source code.
4. the many satellite datas central task stream dispatching algorithm based on nearly data calculating principle according to claim 1, is characterized in that, in step 4, and comprising across computation model automatic compiling of described algorithm:
Step 4-2-1: utilize and be pre-configured in the decompression order matched with the compressed file format of source code and dynamic link library in system platform source code and dynamic link library are decompressed, and the path after dynamic link library decompression is kept in pre-configured temporary file;
Step 4-2-2: after algorithm resource migration to target data center, the path corresponding to dynamic link library file simultaneously also migration will come is added in environmental variance, being added in the dynamic link library file path of recording in step 4-2-1 is updated in the environmental variance of active user, and judge whether there is environmental variance of the same name in configuration file, when there is not environmental variance of the same name, newly-built environmental variance to its assignment;
Step 4-2-3: call pre-configured automatic compiling script, compiles Make File file by the pre-configured algorithm of the ssh agreement invocation target data center pre-set, and drives algorithm resource compilation process to automatically perform;
Step 4-2-4: compile the algorithm resource successfully and generate the executable file matched, increases the new algorithm record in target data CENTER ALGORITHM storehouse by pre-configured database manipulation interface.
5. the many satellite datas central task stream dispatching algorithm based on nearly data calculating principle according to claim 4, is characterized in that, in step 4-2-2, when there is environmental variance of the same name, adding this environmental variance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410851865.9A CN104484230B (en) | 2014-12-31 | 2014-12-31 | More satellite data central task stream dispatching algorithms based on nearly data calculating principle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410851865.9A CN104484230B (en) | 2014-12-31 | 2014-12-31 | More satellite data central task stream dispatching algorithms based on nearly data calculating principle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104484230A true CN104484230A (en) | 2015-04-01 |
| CN104484230B CN104484230B (en) | 2019-03-15 |
Family
ID=52758774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410851865.9A Active CN104484230B (en) | 2014-12-31 | 2014-12-31 | More satellite data central task stream dispatching algorithms based on nearly data calculating principle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104484230B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656799A (en) * | 2017-11-06 | 2018-02-02 | 福建师范大学 | The workflow schedule method of communication and calculation cost is considered under a kind of more cloud environments |
| CN108985709A (en) * | 2018-06-26 | 2018-12-11 | 中国科学院遥感与数字地球研究所 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
| CN111475301A (en) * | 2020-04-09 | 2020-07-31 | 清华大学 | Satellite resource allocation method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315424A (en) * | 2008-07-29 | 2008-12-03 | 中国科学院对地观测与数字地球科学中心 | Multi-satellite remote sensing data integrated parallel ground pretreatment system |
| CN102289540A (en) * | 2011-07-01 | 2011-12-21 | 中国航空工业集团公司科学技术委员会 | Workflow-driven genetic algorithm aviation optimization system orienting to service heterogeneous grid |
| CN104239143A (en) * | 2014-09-17 | 2014-12-24 | 中国科学院遥感与数字地球研究所 | Satellite data receiving task scheduling system based on OSGI (Open Service Gateway Initiative) and workflow and satellite data receiving task scheduling method based on OSGI and workflow |
-
2014
- 2014-12-31 CN CN201410851865.9A patent/CN104484230B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315424A (en) * | 2008-07-29 | 2008-12-03 | 中国科学院对地观测与数字地球科学中心 | Multi-satellite remote sensing data integrated parallel ground pretreatment system |
| CN102289540A (en) * | 2011-07-01 | 2011-12-21 | 中国航空工业集团公司科学技术委员会 | Workflow-driven genetic algorithm aviation optimization system orienting to service heterogeneous grid |
| CN104239143A (en) * | 2014-09-17 | 2014-12-24 | 中国科学院遥感与数字地球研究所 | Satellite data receiving task scheduling system based on OSGI (Open Service Gateway Initiative) and workflow and satellite data receiving task scheduling method based on OSGI and workflow |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656799A (en) * | 2017-11-06 | 2018-02-02 | 福建师范大学 | The workflow schedule method of communication and calculation cost is considered under a kind of more cloud environments |
| CN107656799B (en) * | 2017-11-06 | 2021-06-25 | 福建师范大学 | Workflow scheduling method considering communication and computing cost under multi-cloud environment |
| CN108985709A (en) * | 2018-06-26 | 2018-12-11 | 中国科学院遥感与数字地球研究所 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
| CN111475301A (en) * | 2020-04-09 | 2020-07-31 | 清华大学 | Satellite resource allocation method and device and electronic equipment |
| CN111475301B (en) * | 2020-04-09 | 2021-06-11 | 清华大学 | Satellite resource allocation method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104484230B (en) | 2019-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fujimoto | Research challenges in parallel and distributed simulation | |
| CN103019742B (en) | A kind of real time workshop method in many DSP platform | |
| CN108932588B (en) | Hydropower station group optimal scheduling system with separated front end and rear end and method | |
| CN104536937B (en) | Big data all-in-one machine realization method based on CPU GPU isomeric groups | |
| CN104375806B (en) | A kind of parallel computation component, method and corresponding parallel software development method and system | |
| CN103108031A (en) | Cloud-edge topologies | |
| CN103593192B (en) | A kind of algorithm integration based on SLURM scheduling and evaluating system and method | |
| CN103607466B (en) | A kind of wide-area multi-stage distributed parallel grid analysis method based on cloud computing | |
| CN104598425A (en) | General multiprocessor parallel calculation method and system | |
| CN108123994A (en) | A kind of cloud platform framework towards industrial circle | |
| JP2014525640A (en) | Expansion of parallel processing development environment | |
| CN105843182A (en) | Power dispatching accident handling scheme preparing system and power dispatching accident handling scheme preparing method based on OMS | |
| CN105306557A (en) | Bridge health monitoring system based on cloud platform | |
| CN105069702B (en) | A kind of power grid integrated information processing method | |
| CN101741906A (en) | Grid resource management system supporting HLA distribution interactive simulation and implementation method thereof | |
| CN105094984A (en) | Resource scheduling method and system | |
| CN101937359A (en) | Simulation application-orientated universal extensible computing system | |
| Zhang et al. | Design and implementation of task scheduling strategies for massive remote sensing data processing across multiple data centers | |
| CN104484230A (en) | Multiple satellite data centre workflow scheduling algorithm on basis of near data calculation principle | |
| CN113516331A (en) | Building data processing method and device | |
| CN114897447A (en) | Comprehensive energy cooperative control method and system | |
| CN115525724A (en) | Modeling method and system applied to data warehouse and electronic equipment | |
| CN111951935A (en) | Medical cloud system, method, system and medium for medical big data processing | |
| CN103299277B (en) | Gpu system and processing method thereof | |
| CN104794217A (en) | Tile map data and service updating method and system based on parallel computing mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211103 Address after: No. 19, North Fourth Ring West Road, Haidian District, Beijing 100873 Patentee after: Research Institute of aerospace information innovation, Chinese Academy of Sciences Address before: No. 9 Dengzhuang South Road, Haidian District, Beijing 100094 Patentee before: Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences |
|
| TR01 | Transfer of patent right |