CN108984284A - DAG method for scheduling task and device based on off-line calculation platform - Google Patents
DAG method for scheduling task and device based on off-line calculation platform Download PDFInfo
- Publication number
- CN108984284A CN108984284A CN201810667141.7A CN201810667141A CN108984284A CN 108984284 A CN108984284 A CN 108984284A CN 201810667141 A CN201810667141 A CN 201810667141A CN 108984284 A CN108984284 A CN 108984284A
- Authority
- CN
- China
- Prior art keywords
- task
- node
- dag
- nexus
- task node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of DAG method for scheduling task and device based on off-line calculation platform, method includes: metadata information building multiple tasks node according to the pre-stored data, and according to the dependence between dependence building multiple tasks node between metadata；Topological sorting is carried out to task node, generates DAG task nexus figure；According to triggering to the scheduling request of task node, execute the task of corresponding task node, and in real time modifying DAG task nexus figure corresponding task node Show Styles.The present invention can intuitively show the process of scheduling user task using DAG task nexus figure, realize in real time to the monitoring of task schedule, facilitate and traced using DAG task nexus figure to task abnormity, search task abnormity source.Further, DAG task nexus figure keeps task schedule easier, improves task schedule efficiency.
Description
Technical field
The present invention relates to software fields, and in particular to a kind of DAG method for scheduling task and dress based on off-line calculation platform
It sets.
Background technique
Off-line calculation (Offline calculation) is a kind of data computing technique of data-oriented process field, is
Data are obtained towards batch, bulk transfer data, periodic batch meter count the solution shown according to, data.Off-line calculation
Be can be carried out in mass data complicated batch calculates, the holding time is long, data before the computation just completely in place and will not
It changes, the result that batch calculates can be inquired quickly.At present off-line calculation mostly use greatly Sqoop batch obtain data,
HDFS batch storing data, MapReduce batch calculate data, Hive batch calculates the included task schedule of data, Hadoop
Scheme.With the fast development and application of information technology, the scale of industrial application system is expanded rapidly, and the data generated are mad
Formula explodes.Easily data volume just reaches TB grades or even PB grades, and the processing capacity of traditional computing technique and information system is not
It is able to satisfy.In order to solve this status, the off-line calculation based on big data processing starts vigorously to relevant off-line calculation platform
Development.Before online, by carrying out simulating, verifying inside offline environment, to be imitated to various such as rule, models
The assessment of energy avoids human factor from causing loss etc. caused by inaccuracy.
In the prior art, the task schedule of off-line calculation platform generally uses such as the following processing mode:
1) Chinese patent that number of patent application is 201510080579.1, a kind of method and system of task schedule, mainly
It is to receive the task that operation platform is sent by messaging bus to complete message, updates the follow-up work of current task and current task
Task status；Ready task is obtained, ready task is grouped by task type, override grade is pressed to the ready task in every group
Sequence；Obtain task quantity in the maximum number of concurrent, ready task quantity, operation of every kind of task type；Pass through every kind of task class
The maximum number of concurrent of type, ready task quantity, task quantity in operation, adjust the maximum number of concurrent of every kind of task type, will be every
The ready task of kind task type is distributed to respective operation platform operation by putting in order for ready task in each group respectively, directly
Into the operation of every kind of task type, task quantity reaches the maximum number of concurrent of respective task type.
2) Chinese patent that number of patent application is 201710046973.2, a kind of method for scheduling task and dress of shared cluster
It sets, determines the resource and scheduling parameter in shared cluster；Whether regular check needs Checkpointing；When needing Checkpointing
When, determine the needing Checkpointing of the task, and Checkpointing for it；When there is new task arrival, updates and be currently running
Task with mix waiting list；When the resource behaviour in service of shared cluster changes, updates being currently running for task and mix
Close waiting list.
For above-mentioned processing mode, 1) processing mode in mainly stresses to carry out task by task type, task priority
Distribution, does not focus on the upstream and downstream dependence between task node, when task nexus is more complicated, can not clearly distinguish task
Between dependence, influence the efficiency and accuracy of distribution task.2) Checkpointing is laid particular emphasis in, real-time monitor resource becomes
Change, updates task status.It can however not the state change for providing certain task is caused by being changed as which task (in backtracking
Trip), the state change of task will lead to which task changes information such as (backtracking downstreams), i.e., can not search out and cause
The task source of job change.And above two processing mode all can not clearly show the upstream and downstream between different task node
Dependence is supplied to the visual task processing exhibition method of user.
Summary of the invention
In view of the above problems, it proposes on the present invention overcomes the above problem or at least be partially solved in order to provide one kind
State the DAG method for scheduling task and device based on off-line calculation platform of problem.
According to an aspect of the invention, there is provided a kind of DAG method for scheduling task based on off-line calculation platform, packet
It includes:
Metadata information according to the pre-stored data constructs multiple tasks node, and more according to dependence building between metadata
Dependence between a task node；
Topological sorting is carried out to task node, generates DAG task nexus figure；
According to triggering to the scheduling request of task node, the task of corresponding task node, and real time modifying DAG are executed
The Show Styles of corresponding task node in task nexus figure.
According to another aspect of the present invention, a kind of DAG task scheduling apparatus based on off-line calculation platform is provided, is wrapped
It includes:
Module is constructed, is suitable for metadata information according to the pre-stored data and constructs multiple tasks node, and according between metadata
Dependence constructs the dependence between multiple tasks node；
Generation module is suitable for carrying out topological sorting to task node, generates DAG task nexus figure；
Scheduler module executes the task of corresponding task node suitable for the scheduling request according to triggering to task node,
And in real time modifying DAG task nexus figure corresponding task node Show Styles.
According to another aspect of the invention, provide a kind of electronic equipment, comprising: processor, memory, communication interface and
Communication bus, processor, memory and communication interface complete mutual communication by communication bus；
For memory for storing an at least executable instruction, it is above-mentioned based on off-line calculation that executable instruction executes processor
The corresponding operation of DAG method for scheduling task of platform.
In accordance with a further aspect of the present invention, a kind of computer storage medium is provided, at least one is stored in storage medium
Executable instruction, the DAG method for scheduling task that executable instruction executes processor as above-mentioned based on off-line calculation platform are corresponding
Operation.
The DAG method for scheduling task and device based on off-line calculation platform provided according to the present invention, according to being stored in advance
Metadata information construct multiple tasks node, and according between metadata dependence building multiple tasks node between dependence close
System；Topological sorting is carried out to task node, generates DAG task nexus figure；According to triggering to the scheduling request of task node, hold
The task of the corresponding task node of row, and in real time modifying DAG task nexus figure corresponding task node Show Styles.This hair
The bright process that scheduling user task can intuitively be showed using DAG task nexus figure is realized in real time to task schedule
It monitors, facilitates and task abnormity is traced using DAG task nexus figure, search task abnormity source.Further, DAG task
Relational graph keeps task schedule easier, improves task schedule efficiency.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows the stream of the DAG method for scheduling task according to an embodiment of the invention based on off-line calculation platform
Cheng Tu；
Fig. 2 shows a DAG task nexus diagrams to be intended to；
Fig. 3 shows a task schedule flow diagram；
Fig. 4 shows another DAG task nexus diagram and is intended to；
Fig. 5 shows each feature list schematic diagram of a task schedule；
Fig. 6 shows the DAG method for scheduling task in accordance with another embodiment of the present invention based on off-line calculation platform
Flow chart；
Fig. 7 shows the function of the DAG task scheduling apparatus according to an embodiment of the invention based on off-line calculation platform
It can block diagram；
Fig. 8 shows the structural schematic diagram of a kind of electronic equipment according to an embodiment of the invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
Fig. 1 shows the stream of the DAG method for scheduling task according to an embodiment of the invention based on off-line calculation platform
Cheng Tu.As shown in Figure 1, the DAG method for scheduling task based on off-line calculation platform specifically comprises the following steps:
Step S101, metadata information according to the pre-stored data constructs multiple tasks node, and relies on according between metadata
Relationship constructs the dependence between multiple tasks node.
Metadata information includes such as task names, task status, scheduling type, specified execution equipment, executes the time
Etc. information.Scheduling type includes such as Hive-sql, Shell, data synchronization, Spark-sql, Python, dummy node type.Appoint
Business state include such as normal, initialization, ready, execution queue, operation, successfully, unsuccessfully, freeze state.Believed according to metadata
The task node comprising task names and scheduling type can be generated in breath, and each task node can be set according to task status
Feature.Such as represented using feature #e6f4f8 the task status of task node as normal, feature rgba (0,91,252,
0.4) task that the task status of task node represents task node as initialization, feature rgba (0,91,252,0.6) is represented
State is that ready, feature rgba (0,91,252,0.8) represents the task status of task node to execute queue, feature
Rgba (0,91,252,1) represents the task that the task status of task node represents task node as operation, feature #3EBB44
State be successfully, feature #E93A3A represent the task status of task node as failure, feature #f2f3f7 represent appoint
The task status of business node is to freeze.It is above for example, each data, setting of feature etc. can in metadata information
To be configured according to performance, herein without limitation.After obtaining each task node, it can be closed according to being relied between metadata
System is directed toward the dependence between building multiple tasks node using arrow.Specifically, in view of each task node has dependence
0, the task nodes of one or more upstream-downstream relationships, can be deposited accordingly when metadata information is stored in advance in database
The dependence between each metadata is stored up, according to the dependence between each metadata, using such as breadth first search mode, Ke Yixian
The upstream node or downstream node for finding a task node are saved in the upstream for continuing to search its upstream node or downstream node
Point or downstream node, and so on, until traversal completes the lookup of all task nodes, the dependence for obtaining all task nodes is closed
System.It is directed toward using arrow, the dependence between multiple tasks node can be constructed.As shown in Fig. 2, upstream node passes through arrow
It is directed toward downstream node, there is dependence between two nodes.
Step S102 carries out topological sorting to task node, generates DAG task nexus figure.
Topological sorting is carried out to each task node, and relies on the dependence between each task node, by each task
Dependence between node carries out combing arrangement, so that each task node is attached by dependence, obtains such as Fig. 2 institute
The DAG task nexus figure shown.Wherein, phase between each task node and all task nodes is listed in DAG task nexus figure
Mutual dependence.As first task node be it is current execute task node, there is no a upstream task node, when execution, from
First task node successively executes.Each task node is indicated in Fig. 2 with rectangle, and upper layer text is the task of task node
Title, lower layer's text are the scheduling type of task node, and the task node of different task state is indicated using different features,
Arrow executes direction and shows the dependence between each task node.Different frames or arrow color can be set for each task node
Equal differentiations currently execute in task node and other adjacent nodes etc..When it is implemented, it can also be selected according to performance
DAG task nexus figure is arranged in his mode.
Above two step completes the drafting to DAG task nexus figure.It can be using as schemed to the drafting of DAG task nexus figure
Deployment way shown in 3 stores dependence between metadata information and metadata, by database in the database in advance
Dependence such as is analyzed, is processed at the processing between pre-stored metadata information and metadata, will treated data storage
In shared memory, the data in memory are then obtained by server, is converted, is such as converted to JSON format, then pass through
Network transmission gives each client, carries out the drafting of DAG task nexus figure according to data by each client.
Step S103 executes the task of corresponding task node according to triggering to the scheduling request of task node, and real
The Show Styles of corresponding task node in Shi Xiugai DAG task nexus figure.
When being scheduled processing to task node, the scheduling of task node can be asked with being triggered manually according to user
It asks, executes the task of corresponding task node.As shown in figure 4, user can choose the task node in the lower left corner, by clicking mouse
Right button mode is marked, shows the function menu of the task node.Trip is run down again in reselection function menu, to complete to traverse this
All direct Downstream Jobs nodes of business node.The task status of each Downstream Jobs node is first judged, if certain Downstream Jobs section
The task status of point is initialization, then directly dispatches the Downstream Jobs node.In the task schedule success of the Downstream Jobs node
Afterwards, according to implementing result, the Show Styles of the corresponding task node of real time modifying such as modifies the task status of Downstream Jobs node
For success, feature is revised as #3EBB44, the Show Styles such as corresponding frame, arrow color for modifying former upstream task node,
Mark currently performed task node etc..Further, appoint in all direct downstreams that can also continue to traverse the Downstream Jobs node
Business node, until having traversed all Downstream Jobs nodes etc..By being changed in real time according to the scheduling result for executing task node return
The Show Styles for becoming each task node of DAG task nexus figure, makes it show different features, family can be used intuitively
Which dispatched to task node, which task node task schedule terminates, which task node is dispatched successfully, which is appointed
Node scheduling be engaged in unsuccessfully etc..If a task node scheduling failure, can stop all Downstream Jobs sections to the task node
Point is scheduled, and by DAG task nexus figure, can be intuitively found the source of task abnormity, that is, be dispatched the task section of failure
Point.
Alternatively, when being scheduled processing to task node, it can also be by the system period with calling triggering to task node
Scheduling request, execute the task of corresponding task node, and the task status in metadata information is changed according to implementing result.
Once task status is listened to change, it can be with the corresponding real-time corresponding each task section for changing DAG task nexus figure
The task status of point.Specifically, periodically can be scheduled and be asked to task node by system at set time intervals
It asks.Since the data that project generates after certain interval of time are different, appoint so system needs certain interval of time to reschedule
Business node.To reduce the consumption that human resources are called manually, system can set period Automatic dispatching task.Further, if
After system call, mission failure is executed, the modes such as alarm or transmission mail is can be sent out and notifies operation maintenance personnel, by operation maintenance personnel
By checking DAG task nexus figure, the task node of failure is searched, task abnormity source is found.As shown in figure 3, scheduler can
With pre-stored data in periodically repeating query database, task therein is distributed to actuator, task is executed by actuator.
The implementing result that can be returned according to actuator in corresponding DAG task nexus figure, the corresponding task node of real time modifying and correlation
The Show Styles of task node.
It is scheduled performed function menu for each task node, is referred to shown in Fig. 5, the system period calls institute
The periodic duty function of execution includes such as test, complement evidence, check example, freeze, thawing, recalls upstream, backtracking downstream, is used
The manual task function that family is triggered manually includes such as running.Corresponding task O&M contains period example, manual example, survey
Example is tried, mends and runs example.This four examples have with function: running down trip again, freeze, thaw, be set to function, stopping, running, return again
It traces back upstream, backtracking downstream etc..Periodic duty is periodically scheduled task node according to preset time interval,
Manual task only when triggering task node manually, generates scheduler task.Task schedule generates a task instances each time.
If the every scheduling of periodic duty is primary, a cycle example is generated；Manual task operation is primary, generates a manual example.Test
Example and manual example, which can be manually operated DAG task nexus figure by user, to be generated.
The DAG method for scheduling task based on off-line calculation platform provided according to the present invention, member number according to the pre-stored data
According to information architecture multiple tasks node, and according to the dependence between dependence building multiple tasks node between metadata；It is right
Task node carries out topological sorting, generates DAG task nexus figure；According to triggering to the scheduling request of task node, execution pair
The task for the task node answered, and in real time modifying DAG task nexus figure corresponding task node Show Styles.Benefit of the invention
It can intuitively show the process of scheduling user task with DAG task nexus figure, realize in real time to the monitoring of task schedule,
Facilitate and task abnormity is traced using DAG task nexus figure, searches task abnormity source.Further, DAG task nexus figure
Keep task schedule easier, improves task schedule efficiency.
Fig. 6 shows the DAG method for scheduling task in accordance with another embodiment of the present invention based on off-line calculation platform
Flow chart.As shown in fig. 6, the DAG method for scheduling task based on off-line calculation platform specifically comprises the following steps:
Step S601, metadata information according to the pre-stored data constructs multiple tasks node, and relies on according between metadata
Relationship constructs the dependence between multiple tasks node.
Step S602 carries out topological sorting to task node, generates DAG task nexus figure.
Above step referring to Fig.1 in embodiment step S101-S102 description, details are not described herein.
DAG task nexus figure is arranged in current display screen according to the dimension information of current display screen in step S603
Display position and dimension information, for showing user to check.
The size of the device screen as used in different user is different, to can be adapted for DAG task nexus figure not
Same screen size, makes its generalization, meets the needs of different user.It needs that DAG is arranged according to different screen size informations
The different dimension informations of task nexus figure.Firstly, the dimension information of current display screen is obtained, such as screen width (SW), screen
Highly (SH).And the dimension information that DAG task nexus figure is initial.DAG task nexus figure width (DW), DAG task nexus figure
Highly (DH), DAG task nexus figure initial proportion (initialScale).Calculate the maximum value of (SW-DW) and (SH-DH), root
According to the maximum difference of the high difference of width, amplification/diminution ratio of adjust automatically DAG task nexus figure.It can also be according to the screen of calculating
The high difference of width of curtain and DAG task nexus figure, DAG task nexus figure is half-and-half calculated respectively and shows in current display screen
Position enables placed in the middle in the horizontal vertical of current display screen.Such as DAG task nexus figure ratio Scale=MAX ((SW-
DW), (SH-DH))/initialScale, DAG task nexus figure position Point (x, y)=((SW-DW)/2, (SH-DH)/2).
The above are for example, specific calculation can be arranged according to performance, herein without limitation.
Step S604, according to the operation requests to any task node in DAG task nexus figure that user triggers, modification
The Show Styles of corresponding task node in DAG task nexus figure.
The operation requests that monitoring users trigger the operation of any task node in DAG task nexus figure, specifically, such as
User switches the task node chosen, and the operation requests of triggering are to choose operation requests, can change in real time in DAG task nexus figure
Task node Graphicxtras Frames Collection and arrow color show the task node of switching；Mouse is suspended on task node by user, point
Right mouse button is hit, the operation requests of triggering are to check operation requests, can show the function dish of the task node as shown in Figure 4
It is single.It, can be according to implementing result in DAG task nexus figure after executing task node if user further clicks on and runs function menu again
Change feature, the arrow color etc. of task node in real time；User's roll mouse idler wheel, the operation requests of triggering are zoom operations
It requests, size of the entire DAG task nexus figure with each task node itself can be zoomed in or out in real time in DAG task nexus figure
Information, so as to the relationship clearly checked between local each task node.
Step S605 executes the task of corresponding task node according to triggering to the scheduling request of task node, and real
The Show Styles of corresponding task node in Shi Xiugai DAG task nexus figure.
The step referring to Fig.1 in embodiment step S103 description, details are not described herein.
The DAG method for scheduling task based on off-line calculation platform provided according to the present invention, according to metadata information and member
Dependence constructs DAG task nexus figure between data, clearly describes dependence and mission bit stream between task node, intuitively
The step of showing task schedule and process, real-time monitoring users operation and task status are abnormal, find task abnormity source.Into one
Step adjusts the display of DAG task nexus figure according to different screen dimension information, to meet the device screen demand of different user,
Improve the experience effect that user uses.
Fig. 7 shows the function of the DAG task scheduling apparatus according to an embodiment of the invention based on off-line calculation platform
It can block diagram.As shown in fig. 7, the DAG task scheduling apparatus based on off-line calculation platform includes following module:
Building module 710 is suitable for: metadata information according to the pre-stored data constructs multiple tasks node, and according to metadata
Between dependence building multiple tasks node between dependence.
Generation module 720 is suitable for: carrying out topological sorting to task node, generates DAG task nexus figure.
Scheduler module 730 is suitable for: according to triggering to the scheduling request of task node, executing appointing for corresponding task node
Business, and in real time modifying DAG task nexus figure corresponding task node Show Styles.
Optionally, building module 710 is further adapted for: being generated according to metadata information comprising task names and scheduling type
Task node, and according to task status be arranged task node feature；Referred to according to dependence between metadata using arrow
Dependence between building multiple tasks node.
Optionally, device further include: setup module 740.
Setup module 740 is suitable for: according to the dimension information of current display screen, DAG task nexus figure is arranged and shows currently
Display position and dimension information in display screen curtain, for showing user to check.
Optionally, device further include: operation module 750.
Operation module 750 is suitable for: according to asking to the operation of any task node in DAG task nexus figure for user's triggering
It asks, modifies the Show Styles of corresponding task node in DAG task nexus figure；Wherein, operation requests include choose operation requests,
Operation requests are requested and/or checked to zoom operations；Show Styles include feature, dimension information, task node Graphicxtras Frames Collection and/
Or arrow color.
Optionally, scheduler module 730 is further adapted for: being triggered manually according to user and/or the system period calls triggering ground
To the scheduling request of task node, the task of corresponding task node is executed；According to the implementing result of task node, real time modifying
The Show Styles of corresponding task node and inter-related task node.
The description of step is corresponded in the description reference method embodiment of above each module, details are not described herein.
The DAG task scheduling apparatus based on off-line calculation platform provided according to the present invention, member number according to the pre-stored data
According to information architecture multiple tasks node, and according to the dependence between dependence building multiple tasks node between metadata；It is right
Task node carries out topological sorting, generates DAG task nexus figure；According to triggering to the scheduling request of task node, execution pair
The task for the task node answered, and in real time modifying DAG task nexus figure corresponding task node Show Styles.Benefit of the invention
It can intuitively show the process of scheduling user task with DAG task nexus figure, realize in real time to the monitoring of task schedule,
Facilitate and task abnormity is traced using DAG task nexus figure, searches task abnormity source.Further, DAG task nexus figure
Keep task schedule easier, improves task schedule efficiency.
Present invention also provides a kind of nonvolatile computer storage media, the computer storage medium is stored at least
One executable instruction, the computer executable instructions can be performed in above-mentioned any means embodiment based on off-line calculation platform
DAG method for scheduling task.
Fig. 8 shows the structural schematic diagram of a kind of electronic equipment according to an embodiment of the invention, and the present invention is specifically real
Example is applied not limit the specific implementation of electronic equipment.
As shown in figure 8, the electronic equipment may include: processor (processor) 802, communication interface
(Communications Interface) 804, memory (memory) 806 and communication bus 808.
Wherein:
Processor 802, communication interface 804 and memory 806 complete mutual communication by communication bus 808.
Communication interface 804, for being communicated with the network element of other equipment such as client or other servers etc..
Processor 802 can specifically execute the above-mentioned DAG task tune based on off-line calculation platform for executing program 810
Spend the correlation step in embodiment of the method.
Specifically, program 810 may include program code, which includes computer operation instruction.
Processor 802 may be central processor CPU or specific integrated circuit ASIC (Application
Specific Integrated Circuit), or be arranged to implement the integrated electricity of one or more of the embodiment of the present invention
Road.The one or more processors that electronic equipment includes can be same type of processor, such as one or more CPU；It can also
To be different types of processor, such as one or more CPU and one or more ASIC.
Memory 806, for storing program 810.Memory 806 may include high speed RAM memory, it is also possible to further include
Nonvolatile memory (non-volatile memory), for example, at least a magnetic disk storage.
Program 810 specifically can be used for so that processor 802 execute in above-mentioned any means embodiment based on offline
Calculate the DAG method for scheduling task of platform.The specific implementation of each step may refer to above-mentioned based on off-line calculation platform in program 810
DAG task schedule embodiment in corresponding steps and unit in corresponding description, this will not be repeated here.The technology people of fields
Member can be understood that, for convenience and simplicity of description, the equipment of foregoing description and the specific work process of module, can be with
With reference to the corresponding process description in preceding method embodiment, details are not described herein.
Algorithm and display are not inherently related to any particular computer, virtual system, or other device provided herein.
Various general-purpose systems can also be used together with teachings based herein.As described above, it constructs required by this kind of system
Structure be obvious.In addition, the present invention is also not directed to any particular programming language.It should be understood that can use various
Programming language realizes summary of the invention described herein, and the description done above to language-specific is to disclose this hair
Bright preferred forms.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be practiced without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of the various inventive aspects,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect
Shield the present invention claims features more more than feature expressly recited in each claim.More precisely, such as right
As claim reflects, inventive aspect is all features less than single embodiment disclosed above.Therefore, it then follows tool
Thus claims of body embodiment are expressly incorporated in the specific embodiment, wherein each claim conduct itself
Separate embodiments of the invention.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodiment
Member or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use any
Combination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed
All process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Benefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purpose
It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included certain features rather than other feature, but the combination of the feature of different embodiments mean it is of the invention
Within the scope of and form different embodiments.For example, in detail in the claims, embodiment claimed it is one of any
Can in any combination mode come using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processors
Software module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice
Microprocessor or digital signal processor (DSP) realize the DAG according to an embodiment of the present invention based on off-line calculation platform
The some or all functions of some or all components in task scheduling apparatus.The present invention is also implemented as executing
Some or all device or device programs of method as described herein are (for example, computer program and computer journey
Sequence product).It is such to realize that program of the invention can store on a computer-readable medium, either can have one or
The form of multiple signals.Such signal can be downloaded from an internet website to obtain, be perhaps provided on the carrier signal or
It is provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability
Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch
To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame
Claim.
Claims (12)
1. a kind of DAG method for scheduling task based on off-line calculation platform comprising:
Metadata information according to the pre-stored data constructs multiple tasks node, and constructs multiple according to dependence between metadata
Dependence between business node；
Topological sorting is carried out to task node, generates DAG task nexus figure；
According to triggering to the scheduling request of task node, the task of corresponding task node, and DAG described in real time modifying are executed
The Show Styles of corresponding task node in task nexus figure.
2. according to the method described in claim 1, wherein, the metadata information includes task names, task status, scheduling class
Type, specified execution equipment and/or execution time；
The metadata information according to the pre-stored data constructs multiple tasks node, and more according to dependence building between metadata
Dependence between a task node specifically:
The task node comprising task names and scheduling type is generated according to metadata information, and is arranged according to the task status
The feature of the task node；
The dependence between arrow direction building multiple tasks node is utilized according to dependence between metadata.
3. according to the method described in claim 1, wherein, the method also includes:
According to the dimension information of current display screen, display position of the DAG task nexus figure in current display screen is set
It sets and dimension information, for showing user to check.
4. method according to any one of claim 1-3, wherein the method also includes:
According to the operation requests to any task node in the DAG task nexus figure that user triggers, the DAG task is modified
The Show Styles of corresponding task node in relational graph；Wherein, the operation requests include that operation requests, zoom operations is chosen to ask
Seek and/or check operation requests；The Show Styles includes feature, dimension information, task node Graphicxtras Frames Collection and/or arrow
Color.
5. described according to triggering to the scheduling request of task node according to the method described in claim 1, wherein, execute pair
The task for the task node answered, and in DAG task nexus figure described in real time modifying corresponding task node Show Styles into one
Step includes:
It is triggered manually according to user and/or the system period is with calling triggering to the scheduling request of task node, execute corresponding
The task of business node；
According to the implementing result of the task node, the display sample of the corresponding task node of real time modifying and inter-related task node
Formula.
6. a kind of DAG task scheduling apparatus based on off-line calculation platform comprising:
Module is constructed, is suitable for metadata information according to the pre-stored data and constructs multiple tasks node, and relied on according between metadata
Relationship constructs the dependence between multiple tasks node；
Generation module is suitable for carrying out topological sorting to task node, generates DAG task nexus figure；
Scheduler module executes the task of corresponding task node suitable for the scheduling request according to triggering to task node, and real
The Show Styles of corresponding task node in DAG task nexus figure described in Shi Xiugai.
7. device according to claim 6, wherein the metadata information includes task names, task status, scheduling class
Type, specified execution equipment and/or execution time；
The building module is further adapted for:
The task node comprising task names and scheduling type is generated according to metadata information, and is arranged according to the task status
The feature of the task node；The dependence between arrow direction building multiple tasks node is utilized according to dependence between metadata
Relationship.
8. device according to claim 6, wherein described device further include:
The DAG task nexus figure is arranged in current display screen suitable for the dimension information according to current display screen in setup module
Display position and dimension information in curtain, for showing user to check.
9. device a method according to any one of claims 6-8, wherein described device further include:
Operation module is repaired suitable for the operation requests to any task node in the DAG task nexus figure triggered according to user
Change the Show Styles of corresponding task node in the DAG task nexus figure；Wherein, the operation requests include that operation is chosen to ask
It asks, operation requests are requested and/or checked to zoom operations；The Show Styles includes feature, dimension information, task node frame
Pattern and/or arrow color.
10. device according to claim 6, wherein the scheduler module is further adapted for:
It is triggered manually according to user and/or the system period is with calling triggering to the scheduling request of task node, execute corresponding
The task of business node；According to the implementing result of the task node, the corresponding task node of real time modifying and inter-related task node
Show Styles.
11. a kind of electronic equipment, comprising: processor, memory, communication interface and communication bus, the processor, the storage
Device and the communication interface complete mutual communication by the communication bus；
The memory executes the processor as right is wanted for storing an at least executable instruction, the executable instruction
Ask the corresponding operation of DAG method for scheduling task described in any one of 1-5 based on off-line calculation platform.
12. a kind of computer storage medium, an at least executable instruction, the executable instruction are stored in the storage medium
Processor is set to execute the DAG method for scheduling task according to any one of claims 1 to 5 based on off-line calculation platform corresponding
Operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810667141.7A CN108984284A (en) | 2018-06-26 | 2018-06-26 | DAG method for scheduling task and device based on off-line calculation platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810667141.7A CN108984284A (en) | 2018-06-26 | 2018-06-26 | DAG method for scheduling task and device based on off-line calculation platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108984284A true CN108984284A (en) | 2018-12-11 |
Family
ID=64538283
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810667141.7A Pending CN108984284A (en) | 2018-06-26 | 2018-06-26 | DAG method for scheduling task and device based on off-line calculation platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108984284A (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110245108A (en) * | 2019-07-15 | 2019-09-17 | 北京一流科技有限公司 | It executes body creation system and executes body creation method |
| CN110362315A (en) * | 2019-07-17 | 2019-10-22 | 中国工商银行股份有限公司 | Software systems dispatching method and device based on DAG |
| CN110427252A (en) * | 2019-06-18 | 2019-11-08 | 平安银行股份有限公司 | Method for scheduling task, device and the storage medium of task based access control dependence |
| CN110516971A (en) * | 2019-08-29 | 2019-11-29 | 杭州网易再顾科技有限公司 | Method, apparatus, medium and the calculating equipment of abnormality detection |
| CN110825511A (en) * | 2019-11-07 | 2020-02-21 | 北京集奥聚合科技有限公司 | Operation flow scheduling method based on modeling platform model |
| CN111124646A (en) * | 2019-12-23 | 2020-05-08 | 百度国际科技（深圳）有限公司 | Privatization transformation implementation method and device, electronic equipment and storage medium |
| CN111125444A (en) * | 2019-12-10 | 2020-05-08 | 中国平安财产保险股份有限公司 | Big data task scheduling management method, device, equipment and storage medium |
| CN111147541A (en) * | 2019-11-18 | 2020-05-12 | 广州文远知行科技有限公司 | Node processing method, device and equipment based on parameter server and storage medium |
| CN111221632A (en) * | 2019-10-15 | 2020-06-02 | 中国平安财产保险股份有限公司 | Distributed parallel task scheduling method and device, computer equipment and storage medium |
| CN111522680A (en) * | 2020-04-17 | 2020-08-11 | 支付宝(杭州)信息技术有限公司 | Method, device and equipment for automatically repairing abnormal task node |
| CN112000386A (en) * | 2019-05-08 | 2020-11-27 | 北京奇虎科技有限公司 | Application implementation method and device |
| CN112131297A (en) * | 2020-09-28 | 2020-12-25 | 广州亚信技术有限公司 | Data processing method, device, equipment and storage medium |
| CN112241443A (en) * | 2019-07-16 | 2021-01-19 | 中国移动通信集团浙江有限公司 | Data quality monitoring method and device, computing equipment and computer storage medium |
| CN112418588A (en) * | 2020-09-03 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Task flow display method and device |
| CN112445600A (en) * | 2020-12-15 | 2021-03-05 | 北京首汽智行科技有限公司 | Method and system for issuing offline data processing task |
| CN112749034A (en) * | 2019-10-31 | 2021-05-04 | 北京沃东天骏信息技术有限公司 | Data exception handling method and device, storage medium and electronic equipment |
| CN112764907A (en) * | 2021-01-26 | 2021-05-07 | 网易（杭州）网络有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN112799794A (en) * | 2019-11-14 | 2021-05-14 | 马上消费金融股份有限公司 | Big data scheduling method, device and system and storage device |
| CN113094164A (en) * | 2021-04-27 | 2021-07-09 | 上海商汤科技开发有限公司 | Task processing device and method, computer equipment and storage medium |
| CN113176933A (en) * | 2021-04-08 | 2021-07-27 | 中山大学 | Dynamic cloud network interconnection method for massive workflow tasks |
| CN114489679A (en) * | 2022-02-22 | 2022-05-13 | 北京科杰科技有限公司 | Intelligent analysis system and method for DAG dependency of hadoop big data task |
| CN115421898A (en) * | 2022-11-07 | 2022-12-02 | 杭州比智科技有限公司 | Big data task scheduling management system and method based on quartz framework |
| CN117891584A (en) * | 2024-03-15 | 2024-04-16 | 福建顶点软件股份有限公司 | Task parallelism scheduling method, medium and device based on DAG grouping |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103942034A (en) * | 2014-03-21 | 2014-07-23 | 深圳华大基因科技服务有限公司 | Task scheduling method and electronic device implementing method |
| US20150026691A1 (en) * | 2010-06-25 | 2015-01-22 | Ebay Inc. | Task scheduling based on dependencies and resources |
| CN106293928A (en) * | 2015-06-05 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of overall situation task node dependence method for visualizing, device and system |
| CN106648859A (en) * | 2016-12-01 | 2017-05-10 | 北京奇虎科技有限公司 | Task scheduling method and device |
| CN106815071A (en) * | 2017-01-12 | 2017-06-09 | 上海轻维软件有限公司 | Big data job scheduling system based on directed acyclic graph |
-
2018
- 2018-06-26 CN CN201810667141.7A patent/CN108984284A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150026691A1 (en) * | 2010-06-25 | 2015-01-22 | Ebay Inc. | Task scheduling based on dependencies and resources |
| CN103942034A (en) * | 2014-03-21 | 2014-07-23 | 深圳华大基因科技服务有限公司 | Task scheduling method and electronic device implementing method |
| CN106293928A (en) * | 2015-06-05 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of overall situation task node dependence method for visualizing, device and system |
| CN106648859A (en) * | 2016-12-01 | 2017-05-10 | 北京奇虎科技有限公司 | Task scheduling method and device |
| CN106815071A (en) * | 2017-01-12 | 2017-06-09 | 上海轻维软件有限公司 | Big data job scheduling system based on directed acyclic graph |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000386A (en) * | 2019-05-08 | 2020-11-27 | 北京奇虎科技有限公司 | Application implementation method and device |
| CN110427252A (en) * | 2019-06-18 | 2019-11-08 | 平安银行股份有限公司 | Method for scheduling task, device and the storage medium of task based access control dependence |
| CN110427252B (en) * | 2019-06-18 | 2024-03-26 | 平安银行股份有限公司 | Task scheduling method, device and storage medium based on task dependency relationship |
| CN110245108A (en) * | 2019-07-15 | 2019-09-17 | 北京一流科技有限公司 | It executes body creation system and executes body creation method |
| CN112241443B (en) * | 2019-07-16 | 2023-11-21 | 中国移动通信集团浙江有限公司 | Data quality monitoring method, device, computing equipment and computer storage medium |
| CN112241443A (en) * | 2019-07-16 | 2021-01-19 | 中国移动通信集团浙江有限公司 | Data quality monitoring method and device, computing equipment and computer storage medium |
| CN110362315A (en) * | 2019-07-17 | 2019-10-22 | 中国工商银行股份有限公司 | Software systems dispatching method and device based on DAG |
| CN110362315B (en) * | 2019-07-17 | 2023-05-12 | 中国工商银行股份有限公司 | DAG-based software system scheduling method and device |
| CN110516971A (en) * | 2019-08-29 | 2019-11-29 | 杭州网易再顾科技有限公司 | Method, apparatus, medium and the calculating equipment of abnormality detection |
| CN110516971B (en) * | 2019-08-29 | 2023-04-25 | 杭州网易再顾科技有限公司 | Anomaly detection method, device, medium and computing equipment |
| CN111221632A (en) * | 2019-10-15 | 2020-06-02 | 中国平安财产保险股份有限公司 | Distributed parallel task scheduling method and device, computer equipment and storage medium |
| CN112749034B (en) * | 2019-10-31 | 2024-05-17 | 北京沃东天骏信息技术有限公司 | Data exception processing method and device, storage medium and electronic equipment |
| CN112749034A (en) * | 2019-10-31 | 2021-05-04 | 北京沃东天骏信息技术有限公司 | Data exception handling method and device, storage medium and electronic equipment |
| CN110825511A (en) * | 2019-11-07 | 2020-02-21 | 北京集奥聚合科技有限公司 | Operation flow scheduling method based on modeling platform model |
| CN112799794A (en) * | 2019-11-14 | 2021-05-14 | 马上消费金融股份有限公司 | Big data scheduling method, device and system and storage device |
| CN111147541B (en) * | 2019-11-18 | 2022-11-15 | 广州文远知行科技有限公司 | Node processing method, device and equipment based on parameter server and storage medium |
| CN111147541A (en) * | 2019-11-18 | 2020-05-12 | 广州文远知行科技有限公司 | Node processing method, device and equipment based on parameter server and storage medium |
| CN111125444A (en) * | 2019-12-10 | 2020-05-08 | 中国平安财产保险股份有限公司 | Big data task scheduling management method, device, equipment and storage medium |
| CN111124646A (en) * | 2019-12-23 | 2020-05-08 | 百度国际科技（深圳）有限公司 | Privatization transformation implementation method and device, electronic equipment and storage medium |
| CN111124646B (en) * | 2019-12-23 | 2023-03-14 | 百度国际科技（深圳）有限公司 | Privatization transformation implementation method and device, electronic equipment and storage medium |
| CN111522680A (en) * | 2020-04-17 | 2020-08-11 | 支付宝(杭州)信息技术有限公司 | Method, device and equipment for automatically repairing abnormal task node |
| CN112418588B (en) * | 2020-09-03 | 2023-08-01 | 上海哔哩哔哩科技有限公司 | Task flow display method and device |
| CN112418588A (en) * | 2020-09-03 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Task flow display method and device |
| CN112131297A (en) * | 2020-09-28 | 2020-12-25 | 广州亚信技术有限公司 | Data processing method, device, equipment and storage medium |
| CN112445600A (en) * | 2020-12-15 | 2021-03-05 | 北京首汽智行科技有限公司 | Method and system for issuing offline data processing task |
| CN112764907A (en) * | 2021-01-26 | 2021-05-07 | 网易（杭州）网络有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN112764907B (en) * | 2021-01-26 | 2024-05-10 | 网易（杭州）网络有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN113176933A (en) * | 2021-04-08 | 2021-07-27 | 中山大学 | Dynamic cloud network interconnection method for massive workflow tasks |
| CN113094164A (en) * | 2021-04-27 | 2021-07-09 | 上海商汤科技开发有限公司 | Task processing device and method, computer equipment and storage medium |
| CN114489679A (en) * | 2022-02-22 | 2022-05-13 | 北京科杰科技有限公司 | Intelligent analysis system and method for DAG dependency of hadoop big data task |
| CN115421898A (en) * | 2022-11-07 | 2022-12-02 | 杭州比智科技有限公司 | Big data task scheduling management system and method based on quartz framework |
| CN117891584A (en) * | 2024-03-15 | 2024-04-16 | 福建顶点软件股份有限公司 | Task parallelism scheduling method, medium and device based on DAG grouping |
| CN117891584B (en) * | 2024-03-15 | 2024-05-14 | 福建顶点软件股份有限公司 | Task parallelism scheduling method, medium and device based on DAG grouping |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108984284A (en) | DAG method for scheduling task and device based on off-line calculation platform | |
| Zhang et al. | Flexible job-shop scheduling/rescheduling in dynamic environment: a hybrid MAS/ACO approach | |
| Wong et al. | Integrated process planning and scheduling/rescheduling—an agent-based approach | |
| US20190392369A1 (en) | Cognitive scheduling for cooperative tasks | |
| CN108351805A (en) | Calculate the accelerator processing based on stream of figure | |
| US10547522B2 (en) | Pre-starting services based on traversal of a directed graph during execution of an application | |
| US20170090990A1 (en) | Modeling resource usage for a job | |
| Wu et al. | Real-time load balancing scheduling algorithm for periodic simulation models | |
| Zhu et al. | Scheduling periodical multi-stage jobs with fuzziness to elastic cloud resources | |
| CN106127365A (en) | Quantitative remote sensing On-line Product interactive mode autonomous production method | |
| Klusáček et al. | Efficient grid scheduling through the incremental schedule‐based approach | |
| Kiss et al. | A cloud-agnostic queuing system to support the implementation of deadline-based application execution policies | |
| CN110400085B (en) | Information processing method, information processing apparatus, and information processing system | |
| CN112149838A (en) | Method, device, electronic equipment and storage medium for realizing automatic model building | |
| Ihde et al. | A framework for modeling and executing task-specific resource allocations in business processes | |
| ElMekkawy et al. | Real-time scheduling with deadlock avoidance in flexible manufacturing systems | |
| US20080255908A1 (en) | Generic framework for resource modeling | |
| Entezari-Maleki et al. | Performability-based workflow scheduling in grids | |
| US9195515B1 (en) | Method and system for process load balancing | |
| JP5387083B2 (en) | Job management system and method | |
| Li et al. | A hybrid genetic algorithm and tabu search for multi-objective dynamic jsp | |
| Fakhfakh et al. | Dealing with structural changes on provisioning resources for deadline-constrained workflow | |
| CN114924858A (en) | Task scheduling method and device, storage medium and electronic equipment | |
| Simon et al. | Design of real-time periodic control systems through synchronization and fixed priorities | |
| Hernández et al. | A Simulation-based Scheduling Strategy for Scientific Workflows. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181211 |