CN107783839A - A kind of multi-load data processing method and system - Google Patents
A kind of multi-load data processing method and system Download PDFInfo
- Publication number
- CN107783839A CN107783839A CN201710790538.0A CN201710790538A CN107783839A CN 107783839 A CN107783839 A CN 107783839A CN 201710790538 A CN201710790538 A CN 201710790538A CN 107783839 A CN107783839 A CN 107783839A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- load data
- data
- workflow
- load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/83—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Abstract
The present invention relates to a kind of multi-load data processing method and system, this method to comprise the following steps：Obtain at least one algorithm for handling multi-load data；A workflow is generated according to whole algorithms；Multi-load data are obtained, and determine the data volume of multi-load data；According to default dispatching algorithm and data volume, the calculate node for handling multi-load data is scheduled, it is determined that the calculate node according to workflow processing multi-load data；Workflow and multi-load data are sent to corresponding calculate node, calculate node is handled multi-load data according to workflow.A kind of multi-load data processing method and system provided by the invention, realize while a variety of load datas are handled, realize simultaneously and combined algorithm on demand, can be according to the different data product of a variety of different workflow generations of many algorithms generation, and realize the load balance scheduling of the handling process of high concurrent data.
Description
Technical field
The present invention relates to geoscience computing field, more particularly to a kind of multi-load data processing method and system.
Background technology
At present,, should usually using general high performance satellite ground pretreatment system to the floor treatment of space science data
System uses second task management and dispatching technique, and the first order is the workflow management and scheduling of data processing task, and user is carried
All calculating tasks handed over all are put into request queue, by calling the service for checking credentials to test the request in queue, are passed through
The request of inspection enters scheduling queue, while feeds back to and submit the unique ID of user one of request to be used as mark；The second level is
Subtask manages and scheduling, and the stage receives new scheduler task from first order scheduling, and task is cached using task resource pond,
Then resource is coordinated according to factors such as priority, scheduling strategy, computing resource loads, in terms of determining the subtask which being distributed to
Operator node is handled.
But when handling space science data, above-mentioned data processing scheme has the following disadvantages：
(1) processing of multi-load Data expansion is difficult
It is that the floor treatment of space science data usually requires with traditional remote sensing satellite data handling procedure difference
Tackle the situation that a variety of load imaging mechanisms differ greatly, such as in order to carry out a greater variety of space sections on same flying platform
Experiment and observation are learned, the polytype load including optics, microwave and other Space environment detection classes can be carried, different type carries
The data processing function of lotus can not be extended using the technical scheme.
(2) high concurrent handling process dispatching efficiency is not high
Scheduling of the technical scheme to flow chart of data processing is optimized on the basis of OpenPBS preset schedule algorithms
Realize, when handling the high-speed down data in space science field, the flow chart of data processing dispatching efficiency of high concurrent is limited to
The factors such as task priority, submission order influence, it is impossible to realize the load balancing between calculate node well.
(3) flow chart of data processing can not combine on demand
The flow that in the technical scheme data are carried out with different stage production is all that fixed algoritic module carries out group
Close what is obtained, and in multi-load space science data handling procedure, according to data product production requirement, data product quality point
Analysis demand, it is a necessary functions that combination on demand is carried out to algoritic module, and this is also the function that can not be realized in the technical scheme.
The content of the invention
The technical problems to be solved by the invention are in view of the shortcomings of the prior art, there is provided a kind of multi-load data processing side
Method and system.
The technical scheme that the present invention solves above-mentioned technical problem is as follows：
A kind of multi-load data processing method, comprises the following steps：
Obtain at least one algorithm for handling multi-load data；
One workflow is generated according to all algorithms；
The multi-load data are obtained, and determine the data volume of the multi-load data；
According to default dispatching algorithm and the data volume, the calculate node for handling the multi-load data is carried out
Scheduling, it is determined that according to the workflow processing multi-load data calculate node；
The workflow and the multi-load data are sent to corresponding calculate node, the calculate node is according to
Workflow is handled the multi-load data.
The beneficial effects of the invention are as follows：A kind of multi-load data processing method provided by the invention, by advance according to place
The algorithm generation workflow of multi-load data is managed, solves the problems, such as that the processing of prior art multi-load Data expansion is difficult, realizes
A variety of load datas are handled simultaneously, while realizes and is combined algorithm on demand, can be generated according to many algorithms
The different data product of a variety of different workflow generations, and calculate node is adjusted by default dispatching algorithm
Degree, solves the problems, such as that prior art high concurrent handling process dispatching efficiency is not high, realizes the handling process of high concurrent data
Load balance scheduling.
On the basis of above-mentioned technical proposal, the present invention can also do following improvement.
Further, it is described obtain at least one algorithm for handling multi-load data after, in addition to：
Read the algorithm information of all algorithms；
One XML file is generated according to the algorithm information of all algorithms；
Schema files are obtained, the type of whole values of the XML file is examined according to the Schema files
Survey, when detecting that the type of value of the XML file occurs wrong, testing result is sent to default terminal.
Further, it is described to be specifically included according to each algorithm generation workflow：
OSWorkflow workflow engines are called, at least one algorithm are selected from the XML file, by selection
The algorithm combination is into workflow.
Further, the workflow includes first order data product process, the second DBMS product process, third level number
According to any of product process and fourth stage data product process, wherein：
The first DBMS product process includes：Radiometric calibration algorithm；
The second DBMS product process includes successively：Radiometric calibration algorithm and geometric correction algorithm；
The third level data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm and geometric accurate correction
Algorithm；
The fourth stage data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm, geometric accurate correction are calculated
Method and ortho-rectification algorithm.
Further, in addition to：
The working condition of the calculate node processing multi-load data is monitored, and the working condition is sent out
Give default terminal.
The another technical solution that the present invention solves above-mentioned technical problem is as follows：
A kind of multi-load data handling system, including：
Algorithm acquiring unit, for obtaining at least one algorithm for being used for handling multi-load data；
Flow generation unit, for generating a workflow according to all algorithms；
Data capture unit, for obtaining the multi-load data, and determine the data volume of the multi-load data；
Node scheduling unit, for according to default dispatching algorithm and the data volume, to for handling the multi-load
The calculate node of data is scheduled, it is determined that according to the workflow processing multi-load data calculate node；
Data transmission unit, for the workflow and the multi-load data to be sent into corresponding calculate node, institute
Calculate node is stated to handle the multi-load data according to the workflow.
The beneficial effects of the invention are as follows：A kind of multi-load data handling system provided by the invention, generated by flow single
The algorithm generation workflow of member processing multi-load data, solves the problems, such as that the processing of prior art multi-load Data expansion is difficult, real
Show while a variety of load datas have been handled, while realized and combined algorithm on demand, can be according to many algorithms
The different data product of a variety of different workflow generations of generation, and calculate node is adjusted by node scheduling unit
Degree, solves the problems, such as that prior art high concurrent handling process dispatching efficiency is not high, realizes the handling process of high concurrent data
Load balance scheduling.
Further, the multi-load data handling system also includes：
Algorithm information reading unit, for reading the algorithm information of all algorithms；
Algorithm file generating unit, for generating an XML file according to the algorithm information of all algorithms；
Algorithm file detection unit, for obtaining Schema files, according to the Schema files to the XML file
The type of whole values is detected, and when detecting that the type of value of the XML file occurs wrong, is sent out to default terminal
Send testing result.
Further, the flow generation unit is specifically used for calling OSWorkflow workflow engines, from XML texts
At least one algorithm is selected in part, by the algorithm combination of selection into workflow.
Further, the workflow includes first order data product process, the second DBMS product process, third level number
According to any of product process and fourth stage data product process, wherein：
The first DBMS product process includes：Radiometric calibration algorithm；
The second DBMS product process includes successively：Radiometric calibration algorithm and geometric correction algorithm；
The third level data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm and geometric accurate correction
Algorithm；
The fourth stage data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm, geometric accurate correction are calculated
Method and ortho-rectification algorithm.
Further, in addition to：
Monitoring nodes unit, for being monitored to the working condition of the calculate node processing multi-load data,
And the working condition is sent to default terminal.
The advantages of aspect that the present invention adds, will be set forth in part in the description, and will partly become from the following description
Obtain substantially, or recognized by present invention practice.
Brief description of the drawings
Fig. 1 is a kind of schematic flow sheet of multi-load data processing method provided in an embodiment of the present invention；
Fig. 2 is a kind of schematic flow sheet for multi-load data processing method that another embodiment of the present invention provides；
Fig. 3 is a kind of structural framing figure for multi-load data handling system that another embodiment of the present invention provides.
Embodiment
The principle and feature of the present invention are described below in conjunction with accompanying drawing, the given examples are served only to explain the present invention, and
It is non-to be used to limit the scope of the present invention.
As shown in figure 1, a kind of schematic flow sheet of multi-load data processing method provided in an embodiment of the present invention, this method
Comprise the following steps：
S1, obtain at least one algorithm for handling multi-load data.
Multi-load data refer to the data of the different field in data handling procedure over the ground, for example, multi-load data can
With including optics, microwave with ultraviolet, in another example, multi-load data can include optics and microwave, can also include data over the ground
The data of other field in processing.
Specifically obtaining how many algorithm is solved by the use demand of user, is calculated for example, algorithm can include radiometric calibration
The algorithm that method, geometric correction algorithm, geometric accurate correction algorithm and ortho-rectification algorithm etc. are handled multi-load data.
It should be understood that the algorithm in step S1 refers to handling the core algorithm of multi-load data, in processing multi-load number
According to before, in addition to multi-load data are carried out with the conventional Preprocessing Algorithm such as conventional treatment conversion, be not included in the present embodiment
In described algorithm.
S2, a workflow is generated according to whole algorithms, and purpose is produced according to the different stage data product of different loads
Selected algorithm is different, and the workflow of generation is also different, is one single for example, when only selective radiation calibration algorithm
Workflow, it is a single workflow again when selecting radiometric calibration algorithm and geometric correction algorithm, when selection radiometric calibration
A single workflow again when algorithm, geometric correction algorithm and geometric accurate correction algorithm, when selection radiometric calibration algorithm,
It is a single workflow when geometric correction algorithm, geometric accurate correction algorithm and ortho-rectification algorithm, it is different by these
Workflow is handled multi-load data, generation be different stage product.
It should be noted that for purposes of illustration only, the generation of step S1 and S2 simply to a workflow is illustrated, in fact
During the use of border, magnanimity multi-load data can handle simultaneously, for example, can give birth to simultaneously with parallel generation workflow
The workflow A and B different into two, wherein, workflow A only includes radiometric calibration algorithm, and B includes radiometric calibration algorithm and geometry
Correcting algorithm.
S3, multi-load data are obtained, and determine the data volume of multi-load data, because multi-load data have polytype,
Therefore, data volume here refers to the quantity of all types of multi-load data, so as to subsequent allocations calculate node.
S4, according to default dispatching algorithm and data volume, the calculate node for handling multi-load data is scheduled,
It is determined that the calculate node according to workflow processing multi-load data.
Default dispatching algorithm is that Maui dispatching algorithm is improved by OpenPBS+, including computing resource single node usury
With different scheduling modes such as rate, computing resource overall utilization rate equalization, User Defined scheduling strategies, 500 are achieved over
The load balancing of workflow, more than 200, odd-numbered day processing quantity on order exceedes the concurrent processing flow quantity of more calculate nodes
5000, data sheet daily handling ability can reach 1.8TB.
Because the present embodiment is that a workflow is illustrated, as long as therefore this step distribution respective numbers calculating section
Point, the mode for handling many workflows and batch data, is illustrated below.
For example, it is assumed that the workflow A in step S2 and workflow B are scheduled, it is assumed that workflow A needs number to be processed
It is very big according to measuring, and the data volume very little of workflow B processing, then in order to which overall utilization rate equalizes, many can be calculated
Node distribution gives workflow A, allows these calculate nodes of distribution to be carried out workflow A, these data are handled, will compare
Few calculate node distributes to workflow B, allows the less calculate node of distribution to be carried out workflow B, carrys out processing data, so
It can equalize the overall utilization rate of whole computing cluster, improve the calculating speed of computing cluster.It is appreciated that specific point
With quantity and ratio, it can according to the actual requirements set, will not be repeated here.
It is appreciated that this is a kind of preferable calculate node scheduling mode, in actual applications, there can also be others
Scheduling mode, such as computing resource overall utilization rate equalization, User Defined scheduling strategy, it can so improve system processing
The flexibility of data, improve the efficiency of high concurrent handling process.
S5, workflow and multi-load data are sent to corresponding calculate node, calculate node is according to workflow to overloading
Lotus data are handled.
Calculate node carries out processing calculating after workflow is received, according to workflow to multi-load data, respectively obtains
The data of corresponding level, for example, when workflow is only carries out radiometric calibration algorithm to multi-load data, what is obtained is a series
According to.
A kind of multi-load data processing method that the present embodiment provides, by advance according to the algorithm of processing multi-load data
Workflow is generated, solves the problems, such as that the processing of prior art multi-load Data expansion is difficult, realizes while to a variety of load datas
Handled, while realize and combined algorithm on demand, a variety of different workflows that can be generated according to many algorithms
Different data products is generated, and calculate node is scheduled by default dispatching algorithm, solves prior art height
The problem of concurrent processing flow scheduling is inefficient, realize the load balance scheduling of the handling process of high concurrent data.
As shown in Fig. 2 a kind of schematic flow sheet of the multi-load data processing method provided for another embodiment of the present invention,
This method comprises the following steps：
S1, obtain at least one algorithm for handling multi-load data.
S2, obtained algorithm is detected, mainly the value type of algorithm detected, detect errorless, just meeting
Continue next step, after detecting mistake, the mistake that can dish out simultaneously exits flow.
Preferably, step S2 can be refined as following steps.
S21, read the algorithm information of whole algorithms.Algorithm information is obtained from the algorithm table in external data base,
Including：The publicly-owned parameter of algorithm title, algorithm, the privately owned parameter of algorithm, algorithm are located at the contents such as disk storage position.
After acquisition algorithm information, algorithm information can be subjected to interim storage, so as to subsequent calls.
S22, an XML file is generated according to the algorithm information of whole algorithms, for example, web Service interface can be passed through
Generation.
S23, Schema files are obtained, the type of whole values of XML file is detected according to Schema files, when
When detecting that the type of the value of XML file occurs wrong, testing result is sent to default terminal, for example, can be to tag name
Claim to be detected with key messages such as data types.
S3, OSWorkflow workflow engines are called, at least one calculation is selected from the algorithm list described in XML file
Method, by the algorithm combination of selection into workflow, different according to selected algorithm, the workflow of generation is also different.
It should be noted that due in actual use, many algorithms can be generated according to a variety of production requirements simultaneously,
Therefore, step S3 specially selects all algorithms from XML file, will selected after OSWorkflow workflow engines are called
All algorithms selected are according to sequential combination specified in XML into workflow.
Preferably, a kind of preferred implementation scheme for generating workflow is given below.
Workflow can include the first DBMS product process, the second DBMS product process, third level data generation stream
Any of journey and fourth stage data product process, wherein：
First DBMS product process includes：Radiometric calibration algorithm, what it is by the generation of the first DBMS product process is one
DBMS product.
Specifically, radiant correction model is designed according to view data physics attenuation process in road radiation transmission process, passed through
Gain realizes that radiant correction is handled with deviant, obtains 1 DBMS product.
Second DBMS product process includes successively：Radiometric calibration algorithm and geometric correction algorithm, pass through the second DBMS
Product process generation is secondary data product.
Specifically, the level one data product generated according to radiometric calibration algorithm, tight imaging geometry model or logical is passed through
With the calibration of imaging geometry model realization earth surface deformation, processing obtains secondary data product.
Third level data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm and geometric accurate correction algorithm,
What it is by the generation of third level data product process is three-level data product.
Specifically, the secondary data product generated according to geometric correction algorithm, is entered by ground control point to ground deformation
The accurate correction of row, further improves earth observation data image correction accuracy, and processing obtains three-level data product.
Fourth stage data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm, geometric accurate correction algorithm and
Ortho-rectification algorithm, what it is by the generation of fourth stage data product process is level Four data product.
Specifically, the three-level data product generated according to geometric accurate correction algorithm, passes through DEM (Digital Elevation
Model, digital elevation model) further accurate correction of the realization to ground deformation in image, handle and obtain level Four data product.
S4, multi-load data are obtained, and determine the data volume of multi-load data, because multi-load data have polytype,
Therefore, data volume here refers to the quantity of all types of multi-load data, so as to subsequent allocations calculate node.
S5, according to default dispatching algorithm and data volume, the calculate node for handling multi-load data is scheduled,
It is determined that the calculate node according to workflow processing multi-load data.
S6, workflow and multi-load data are sent to corresponding calculate node, calculate node is according to workflow to overloading
Lotus data are handled.
S7, the working condition for handling calculate node multi-load data are monitored, and working condition are sent to default
Terminal, in actual use, working condition can be fed back to OSWorkflow workflow engines, worked by OSWorkflow
Default terminal is sent to after stream engine package interface information, so as to user monitoring data handling procedure.Monitoring be it is dynamic in real time,
Can be with Millisecond response time interval dynamic access working condition, and the status information in implementation procedure is formatted defeated
Go out.
For example, working condition can include：In not actuated, operation, pause, normal termination, various states are exited etc. extremely,
Can be when algorithm start, workflow ID caused by acquisition algorithm startup, according to work shapes of the workflow ID to each workflow
State is monitored.
Preferably, can also comprise the following steps：
The control instruction of user's input is obtained, calculate node is cancelled according to control instruction, suspends, restart,
The operation such as deletion, the running status for instantaneity control workflow.
As shown in figure 3, a kind of structural framing figure of the multi-load data handling system provided for another embodiment of the present invention,
The system is designed using C/S model, and whole system is realized based on WebService, OSWorkflow, OpenPBS, is mainly included
Client and service end, wherein, client is the operation interface of system, for management algorithm, workflow, monitoring calculating process
Realized Deng, service end based on the high performance computing cluster 11 for being easy to extension, with high performance parallel file system, parallel computation ring
Border etc. is running environment, using task management control as the Scheduling Core of whole system, manages different loads flow chart of data processing.
The structure of client is illustrated first below, client includes：
Algorithm acquiring unit 1, for obtaining at least one algorithm for being used for handling multi-load data.
Algorithm information reading unit 2, for reading the algorithm information of whole algorithms.
Algorithm file generating unit 3, for generating an XML file according to the algorithm information of whole algorithms.
Algorithm file detection unit 4, for obtaining Schema files, the whole of XML file is taken according to Schema files
The type of value is detected, and when detecting that the type of value of XML file occurs wrong, detection knot is sent to default terminal 10
Fruit.
Flow generation unit 5, for calling OSWorkflow workflow engines, at least one calculation is selected from XML file
Method, by the algorithm combination of selection into workflow.
Preferably, workflow includes first order data product process, the second DBMS product process, the generation of third level data
Any of flow and fourth stage data product process, wherein：
First DBMS product process includes：Radiometric calibration algorithm.
Second DBMS product process includes successively：Radiometric calibration algorithm and geometric correction algorithm.
Third level data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm and geometric accurate correction algorithm.
Fourth stage data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm, geometric accurate correction algorithm and
Ortho-rectification algorithm.
Data capture unit 6, for obtaining multi-load data, and determine the data volume of multi-load data.
Node scheduling unit 7, for according to default dispatching algorithm and data volume, to based on handling multi-load data
Operator node is scheduled, it is determined that the calculate node according to workflow processing multi-load data.
Data transmission unit 8, for workflow and multi-load data to be sent into corresponding calculate node.
Monitoring nodes unit 9, the working condition for handling calculate node multi-load data are monitored, and by work
State is sent to default terminal 10.
Next the structure of service end is illustrated, service end includes：
The high performance computing cluster 11 of expandable type, including multiple calculate nodes, calculate node are used for according to workflow pair
Multi-load data are handled.
Reader should be understood that in the description of this specification, reference term " one embodiment ", " some embodiments ", " show
The description of example ", " specific example " or " some examples " etc. mean to combine the specific features of the embodiment or example description, structure,
Material or feature are contained at least one embodiment or example of the present invention.In this manual, above-mentioned term is shown
The statement of meaning property need not be directed to identical embodiment or example.Moreover, specific features, structure, material or the feature of description
It can be combined in an appropriate manner in any one or more embodiments or example.In addition, in the case of not conflicting, this
The technical staff in field can be by the different embodiments or example described in this specification and the spy of different embodiments or example
Sign is combined and combined.
It is apparent to those skilled in the art that for convenience of description and succinctly, the dress of foregoing description
The specific work process with unit is put, the corresponding process in preceding method embodiment is may be referred to, will not be repeated here.
In several embodiments provided herein, it should be understood that disclosed apparatus and method, it can be passed through
Its mode is realized.For example, device embodiment described above is only schematical, for example, the division of unit, is only
A kind of division of logic function, can there is an other dividing mode when actually realizing, for example, multiple units or component can combine or
Person is desirably integrated into another system, or some features can be ignored, or does not perform.
The unit illustrated as separating component can be or may not be physically separate, be shown as unit
Part can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple networks
On unit.Some or all of unit therein can be selected to realize the mesh of scheme of the embodiment of the present invention according to the actual needs
's.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, can also
It is that unit is individually physically present or two or more units are integrated in a unit.It is above-mentioned integrated
Unit can both be realized in the form of hardware, can also be realized in the form of SFU software functional unit.
If integrated unit is realized in the form of SFU software functional unit and is used as independent production marketing or in use, can
To be stored in a computer read/write memory medium.Based on such understanding, technical scheme substantially or
Say that the part to be contributed to prior art, or all or part of the technical scheme can be embodied in the form of software product
Out, the computer software product is stored in a storage medium, including some instructions are causing a computer equipment
(can be personal computer, server, or network equipment etc.) performs all or part of each embodiment method of the present invention
Step.And foregoing storage medium includes：USB flash disk, mobile hard disk, read-only storage (ROM, Read-OnlyMemory), deposit at random
Access to memory (RAM, RandomAccessMemory), magnetic disc or CD etc. are various can be with the medium of store program codes.
More than, it is only embodiment of the invention, but protection scope of the present invention is not limited thereto, and it is any to be familiar with
Those skilled in the art the invention discloses technical scope in, various equivalent modifications or substitutions can be readily occurred in,
These modifications or substitutions should be all included within the scope of the present invention.Therefore, protection scope of the present invention should be wanted with right
The protection domain asked is defined.
Claims (10)
1. a kind of multi-load data processing method, it is characterised in that comprise the following steps：
Obtain at least one algorithm for handling multi-load data；
One workflow is generated according to all algorithms；
The multi-load data are obtained, and determine the data volume of the multi-load data；
According to default dispatching algorithm and the data volume, the calculate node for handling the multi-load data is adjusted
Degree, it is determined that according to the workflow processing multi-load data calculate node；
The workflow and the multi-load data are sent to corresponding calculate node, the calculate node is according to the work
Stream is handled the multi-load data.
2. multi-load data processing method according to claim 1, it is characterised in that described to obtain for handling multi-load
After at least one algorithm of data, in addition to：
Read the algorithm information of all algorithms；
One XML file is generated according to the algorithm information of all algorithms；
Schema files are obtained, the type of whole values of the XML file is detected according to the Schema files, when
When detecting that the type of the value of the XML file occurs wrong, testing result is sent to default terminal.
3. multi-load data processing method according to claim 2, it is characterised in that described to be generated according to each algorithm
Workflow specifically includes：
OSWorkflow workflow engines are called, at least one algorithm are selected from the XML file, by described in selection
Algorithm combination is into workflow.
4. multi-load data processing method according to any one of claim 1 to 3, it is characterised in that the workflow
Generated including the first DBMS product process, the second DBMS product process, third level data product process and fourth stage data
Any of flow, wherein：
The first DBMS product process includes：Radiometric calibration algorithm；
The second DBMS product process includes successively：Radiometric calibration algorithm and geometric correction algorithm；
The third level data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm and geometric accurate correction algorithm；
The fourth stage data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm, geometric accurate correction algorithm and
Ortho-rectification algorithm.
5. multi-load data processing method according to claim 4, it is characterised in that also include：
The working condition of the calculate node processing multi-load data is monitored, and the working condition is sent to
Default terminal.
A kind of 6. multi-load data handling system, it is characterised in that including：
Algorithm acquiring unit, for obtaining at least one algorithm for being used for handling multi-load data；
Flow generation unit, for generating a workflow according to all algorithms；
Data capture unit, for obtaining the multi-load data, and determine the data volume of the multi-load data；
Node scheduling unit, for according to default dispatching algorithm and the data volume, to for handling the multi-load data
Calculate node be scheduled, it is determined that according to the workflow processing multi-load data calculate node；
Data transmission unit, for the workflow and the multi-load data to be sent into corresponding calculate node, the meter
Operator node is handled the multi-load data according to the workflow.
7. multi-load data handling system according to claim 6, it is characterised in that also include：
Algorithm information reading unit, for reading the algorithm information of all algorithms；
Algorithm file generating unit, for generating an XML file according to the algorithm information of all algorithms；
Algorithm file detection unit, for obtaining Schema files, the whole according to the Schema files to the XML file
The type of value is detected, and when detecting that the type of value of the XML file occurs wrong, is sent and is examined to default terminal
Survey result.
8. multi-load data handling system according to claim 7, it is characterised in that the flow generation unit is specifically used
In calling OSWorkflow workflow engines, at least one algorithm is selected from the XML file, by the calculation of selection
Method is combined into workflow.
9. the multi-load data handling system according to any one of claim 7 to 8, it is characterised in that the workflow
Generated including the first DBMS product process, the second DBMS product process, third level data product process and fourth stage data
Any of flow, wherein：
The first DBMS product process includes：Radiometric calibration algorithm；
The second DBMS product process includes successively：Radiometric calibration algorithm and geometric correction algorithm；
The third level data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm and geometric accurate correction algorithm；
The fourth stage data product process includes successively：Radiometric calibration algorithm, geometric correction algorithm, geometric accurate correction algorithm and
Ortho-rectification algorithm.
10. multi-load data handling system according to claim 9, it is characterised in that also include：
Monitoring nodes unit, for being monitored to the working condition of the calculate node processing multi-load data, and will
The working condition is sent to default terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710790538.0A CN107783839A (en) | 2017-09-05 | 2017-09-05 | A kind of multi-load data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710790538.0A CN107783839A (en) | 2017-09-05 | 2017-09-05 | A kind of multi-load data processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107783839A true CN107783839A (en) | 2018-03-09 |
Family
ID=61438186
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710790538.0A Pending CN107783839A (en) | 2017-09-05 | 2017-09-05 | A kind of multi-load data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107783839A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110047049A (en) * | 2019-04-19 | 2019-07-23 | 西安航天恒星科技实业(集团)有限公司 | The compatible real-time quick radiation correction method of more stars |
| CN111813453A (en) * | 2020-04-30 | 2020-10-23 | 中科院计算所西部高等技术研究院 | Calculation board card with OODA (on-off-the-digital-analog) multiprocessor |
| CN113538081A (en) * | 2020-04-21 | 2021-10-22 | 中移(上海)信息通信科技有限公司 | Mall order system and processing method for realizing resource adaptive scheduling |
| CN113778643A (en) * | 2021-08-13 | 2021-12-10 | 中电科芜湖通用航空产业技术研究院有限公司 | Unmanned aerial vehicle task management computer software architecture system and configuration switching method |
| CN115994190A (en) * | 2023-03-23 | 2023-04-21 | 中国科学院空间应用工程与技术中心 | Distributed real-time processing method and system for aerospace mass data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070067373A1 (en) * | 2003-11-03 | 2007-03-22 | Steven Higgins | Methods and apparatuses to provide mobile applications |
| CN102521687A (en) * | 2011-12-01 | 2012-06-27 | 中国资源卫星应用中心 | Miniaturized universal platform for preprocessing remote-sensing satellite data |
-
2017
- 2017-09-05 CN CN201710790538.0A patent/CN107783839A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070067373A1 (en) * | 2003-11-03 | 2007-03-22 | Steven Higgins | Methods and apparatuses to provide mobile applications |
| CN102521687A (en) * | 2011-12-01 | 2012-06-27 | 中国资源卫星应用中心 | Miniaturized universal platform for preprocessing remote-sensing satellite data |
Non-Patent Citations (1)
| Title |
|---|
| 李景山等: "基于工作流的新一代多卫星地面预处理系统设计与实现", 《遥感技术与应用》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110047049A (en) * | 2019-04-19 | 2019-07-23 | 西安航天恒星科技实业(集团)有限公司 | The compatible real-time quick radiation correction method of more stars |
| CN113538081A (en) * | 2020-04-21 | 2021-10-22 | 中移(上海)信息通信科技有限公司 | Mall order system and processing method for realizing resource adaptive scheduling |
| CN113538081B (en) * | 2020-04-21 | 2024-03-22 | 中移(上海)信息通信科技有限公司 | Mall order system and processing method for realizing resource self-adaptive scheduling |
| CN111813453A (en) * | 2020-04-30 | 2020-10-23 | 中科院计算所西部高等技术研究院 | Calculation board card with OODA (on-off-the-digital-analog) multiprocessor |
| CN113778643A (en) * | 2021-08-13 | 2021-12-10 | 中电科芜湖通用航空产业技术研究院有限公司 | Unmanned aerial vehicle task management computer software architecture system and configuration switching method |
| CN115994190A (en) * | 2023-03-23 | 2023-04-21 | 中国科学院空间应用工程与技术中心 | Distributed real-time processing method and system for aerospace mass data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107783839A (en) | A kind of multi-load data processing method and system | |
| Wang et al. | Parallel-machine rescheduling with job unavailability and rejection | |
| Assarzadegan et al. | Minimizing sum of the due date assignment costs, maximum tardiness and distribution costs in a supply chain scheduling problem | |
| CN105929690B (en) | A kind of Flexible Workshop Robust Scheduling method based on decomposition multi-objective Evolutionary Algorithm | |
| Modrák et al. | Flow shop scheduling algorithm to minimize completion time for n-jobs m-machines problem | |
| Shadkam et al. | Multi-objective simulation optimization for selection and determination of order quantity in supplier selection problem under uncertainty and quality criteria | |
| CN110990121B (en) | Kubernetes scheduling strategy based on application portraits | |
| CN106844027A (en) | A kind of method for scheduling task based on node load | |
| CN106022631B (en) | Index weight analysis method | |
| Pastor | LB-ALBP: the lexicographic bottleneck assembly line balancing problem | |
| Rustogi et al. | Parallel machine scheduling: Impact of adding extra machines | |
| Rebai et al. | Scheduling jobs and maintenance activities on parallel machines | |
| CN110119876A (en) | Worksheet method and device | |
| Lobo et al. | An effective lower bound on Lmax in a worker-constrained job shop | |
| CN106934537A (en) | The sub- time limit based on the scheduling of reverse operation stream obtains optimization method | |
| Pooranian et al. | Independent task scheduling in grid computing based on queen bee algorithm | |
| Weng et al. | Control methods for dynamic time-based manufacturing under customized product lead times | |
| Baykasoğlu et al. | Genetic programming based data mining approach to dispatching rule selection in a simulated job shop | |
| Cinar et al. | A taxonomy for the flexible job shop scheduling problem | |
| Mandal et al. | Incorporating dynamism in traditional machine loading problem: an AI-based optimisation approach | |
| Friese et al. | Towards efficient resource allocation for distributed workflows under demand uncertainties | |
| Li et al. | Minimizing job completion time variance for service stability on identical parallel machines | |
| CN110349051A (en) | A kind of grid equipment NQI big data system | |
| CN112948115B (en) | Cloud workflow scheduler pressure prediction method based on extreme learning machine | |
| Cassettari et al. | An healthcare process reengineering using discrete event simulation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180309 |