CN107133089A - A kind of task scheduling server and method for scheduling task - Google Patents
A kind of task scheduling server and method for scheduling task Download PDFInfo
- Publication number
- CN107133089A CN107133089A CN201710287419.3A CN201710287419A CN107133089A CN 107133089 A CN107133089 A CN 107133089A CN 201710287419 A CN201710287419 A CN 201710287419A CN 107133089 A CN107133089 A CN 107133089A
- Authority
- CN
- China
- Prior art keywords
- subtask
- task
- implementing result
- request
- performs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention provides a kind of task scheduling server, and the task scheduling server includes：Judge module, acquisition module, processing module.Judge module, for judging whether current subtask is individually performed by corresponding subsystem.Acquisition module, for when current subtask is individually performed by corresponding subsystem, obtaining the corresponding mission script in subtask.Processing module, for performing mission script, corresponding subsystem is sent to by the execution request of subtask, and monitors the implementing result of subtask, determines whether the subtask performs completion according to the implementing result.The present invention also provides a kind of method for scheduling task.The task scheduling server and method for scheduling task that the present invention is provided can be by calling the mission script of subtask, it is determined that whether current subtask runs succeeded, it is ensured that all subtasks can be performed in order.
Description
Technical field
The present invention relates to the communications field, more specifically to a kind of task scheduling server and method for scheduling task.
Background technology
Directed acyclic graph (DAG, Directed Acyclic Graph) task in big data platform task scheduling system
Scheduling flow is safeguarded that the task scheduling flow based on DAG figures is converted to using workflow schedule instrument by single system
The script that can be recognized by instrument stream scheduler, for example, the task scheduling flow based on DAG figures is turned using Airflow instruments
The dag.py scripts that can be recognized by Airflow are changed to, wherein, Airflow is a workflow open source projects, and dag.py refers to use
The coding script that Python writes.Airflow is that can directly perform task in itself, such as Hive SQL tasks, the resistance between task
It is to be determined whether to perform next subtask according to subtask implementing result to fill in execution, it is ensured that subtask can be performed in order, but
It is the task of some current big datas, such as Hive SQL tasks are completed by single subsystem, so, in Airflow neutrons
Task is changed into the execution of hypertext transfer protocol requests subsystem, the step of subsystem subtasking two, and hyper text protocol is asked successfully
It can not judge whether subsystem performs subtask, hyper text protocol asks successfully to cannot function as subtask really to run succeeded
State, prior art not can determine that whether subtask runs succeeded.
The content of the invention
In consideration of it, the invention provides a kind of task scheduling server and method for scheduling task, the task scheduling service
Device and method for scheduling task can determine that whether subtask runs succeeded, and improve the efficiency of task scheduling.
The present invention provides a kind of task scheduling server, and the task scheduling server includes：
Judge module, for judging whether current subtask is individually performed by corresponding subsystem；
Acquisition module, for when current subtask is individually performed by corresponding subsystem, obtaining the subtask pair
The mission script answered；
Processing module, for performing the mission script, the execution request of the subtask is sent to described corresponding
Subsystem, and the implementing result of the subtask is monitored, determine whether the subtask performs completion according to the implementing result.
Alternatively, the mission script includes first task node, the second task node, wherein, the first task section
Point definition performs request, described to perform request for asking the subtask system to perform the current subtask, described second
Task node definition monitors request, and the monitoring request is used to monitor holding for the subtask system execution current subtask
Row result.
Alternatively, described perform asks to include hypertext transfer protocol requests, and the monitoring request includes polling request, institute
Stating implementing result includes performing completion or executive termination or performing cancelling or performing exception.
Alternatively, the processing module includes：
Performing module, please for performing request to subtask system transmission is described according to the first task node
The subtask system is asked to perform the current subtask；
Monitoring modular, is asked, prison for sending the monitoring to the subtask system according to second task node
The implementing result that the subtask system performs the current subtask is surveyed, if the implementing result is to run succeeded, it is determined that
The subtask runs succeeded, if the implementing result is executive termination or performs cancellation or perform exception, it is determined that
The subtask performs failure.
Alternatively,
The performing module, is additionally operable to perform the first task node of the mission script, according to described first
Business node sends hypertext transfer protocol requests to the subtask system；
The monitoring modular, is additionally operable to perform second task node of the mission script according to second task
Node sends polling request to the subtask system, and subtask system described in periodic monitor performs holding for the current subtask
Row result, if the implementing result is to run succeeded, it is determined that the subtask runs succeeded, if the implementing result is execution
Terminate or perform and cancel or perform exception, it is determined that the subtask performs failure, if the implementing result is execution
When success, executive termination or execution are cancelled or perform abnormal, then stop the transmission polling request, if described perform knot
Fruit is space-time, then is spaced preset time and sends the polling request.
Present invention also offers a kind of method for scheduling task, the method for scheduling task is applied to task scheduling server,
The method for scheduling task includes：
Judge whether current subtask is individually performed by corresponding subsystem；
When current subtask is individually performed by corresponding subsystem, the corresponding mission script in the subtask is obtained；
The mission script is performed, the execution request of the subtask the corresponding subsystem is sent to, and monitor
The implementing result of the subtask, determines whether the subtask performs completion according to the implementing result.
Alternatively, the mission script includes first task node, the second task node, wherein, the first task section
Point definition performs request, described to perform request for asking the subtask system to perform the current subtask, described second
Task node definition monitors request, and the monitoring request is used to monitor holding for the subtask system execution current subtask
Row result.
Alternatively, described perform asks to include hypertext transfer protocol requests, and the monitoring request includes polling request, institute
Stating implementing result includes performing completion or executive termination or performing cancelling or performing exception.
Alternatively, the mission script is performed, the execution request of the subtask is sent to the corresponding subsystem,
And the implementing result of the subtask is monitored, determine whether the subtask performs completion according to the implementing result, including：
Request is performed to subtask system transmission is described according to second task node, the subtask system is asked
System performs the current subtask；
The monitoring is sent to the subtask system to ask, monitor the subtask system according to second task node
System performs the implementing result of the current subtask；
If the implementing result is to run succeeded, it is determined that the subtask runs succeeded；
If the implementing result is executive termination or performs cancellation or perform exception, it is determined that held the subtask
Row failure.
Alternatively, it is described to perform the mission script, the execution request of the subtask is sent to the corresponding son
System, and the implementing result of the subtask is monitored, determine whether the subtask performs completion according to the implementing result, wrap
Include：
The first task node of the mission script, second task node are performed, according to the first task
Node sends hypertext transfer protocol requests to the subtask system；
Polling request, subtask system described in periodic monitor are sent to the subtask system according to second task node
System performs the implementing result of the current subtask；
If the implementing result is to run succeeded, it is determined that the subtask runs succeeded；
If the implementing result is executive termination or performs cancellation or perform exception, it is determined that held the subtask
Row failure；
If the implementing result is runs succeeded, executive termination or perform when cancelling or performing abnormal, stop
Send the polling request；
If the implementing result is space-time, interval preset time sends the polling request.
Whether the task scheduling server and method for scheduling task that the present invention is provided can judge current subtask by corresponding
Subsystem is individually performed, when current subtask is individually performed by corresponding subsystem, obtains corresponding of the subtask
Business script, performs the mission script, the execution request of the subtask is sent into the corresponding subsystem, and monitor institute
The implementing result of subtask is stated, determines whether the subtask performs completion according to the implementing result, so, can determine that son is appointed
Whether business is run succeeded, it is ensured that all subtasks can be performed in order, improves the efficiency of task scheduling.
Brief description of the drawings
Fig. 1 is realizes the layering schematic diagram of the big data platform where each of the invention embodiment task scheduling server；
The application environment schematic diagram of big data platforms of the Fig. 2 where task scheduling server as shown in Figure 1；
Fig. 3 is the functional block diagram of the task scheduling server of the embodiment of the present invention one；
Fig. 4 is the functional block diagram of the task scheduling server of the embodiment of the present invention two；
Fig. 5 is the application environment schematic diagram of the task scheduling server of the embodiment of the present invention three；
Fig. 6 is the flow chart of the method for scheduling task of the embodiment of the present invention four；
Fig. 7 is the flow chart of the method for scheduling task of the embodiment of the present invention five；
Fig. 8 is the flow chart of the method for scheduling task of the embodiment of the present invention six；
Fig. 9 is the code schematic diagram of the task node of the embodiment of the present invention 7 second.
The realization, functional characteristics and advantage of the object of the invention will be described further referring to the drawings in conjunction with the embodiments.
Embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
The optional task scheduling server of each embodiment one of the invention is realized referring now to accompanying drawing description.Follow-up
In description, using the suffix of such as " module ", " part " or " unit " for representing element only for be conducive to the present invention
Illustrate itself do not have specific meaning.Therefore, " module " can be used mixedly with " part ".
Task scheduling server can be implemented in a variety of manners.For example, the server described in the present invention can include
Such as large server, small server.
Fig. 1 is realizes the layering schematic diagram of the big data platform where each of the invention embodiment task scheduling server.
Fig. 1 includes data analysis layer 20, programming model layer 30, data storage layer 40, File Store layer 50, data set stratification 60.Number
Include the data of application data, database, daily record, other data sources according to the integrated data of integrated layer 60, File Store layer is used for
Various documentations are stored, data storage layer 40 is used to store metadata, and programming model layer 30 is used to carry out according to specific algorithm
Model is programmed, and data analysis layer 20 is used to be made a concrete analysis of for data, services, the cores of data, services be distributed storage,
Real-time calculating, off-line calculation etc..
The application environment schematic diagram of big data platforms of the Fig. 2 where task scheduling server as shown in Figure 1.Fig. 2
It is laid out, is calculated in fig. 2 including task call server 10, data storage server 270, data under Fig. 1 layering
Server 271, subsystem 272, task call server 10 is respectively at data storage server 270, data calculation server
271, subsystem 272 is communicatively coupled.Task call server 10 can be calculated according to data storage server 270, data and taken
The analysis of business device 271, which is calculated, obtains various subtasks, and task call server 10 is obtained after subtask, asks corresponding subsystem
System 272 performs corresponding subtask.
Various servers have multiple in supplementary notes, big data, one are only schematically drawn herein, in this reality
Apply in example and the number of server is not limited.
Task scheduling server 10 can include, but not limited to memory, controller, wireless communication unit, input list
Member, input block, interface unit and power subsystem.It should be understood, however, that being not required for implementing all component.It can substitute
Implement more or less components in ground.The element of task scheduling server 10 will be discussed in more detail below.
Wireless communication unit generally includes one or more assemblies, and it allows task scheduling server 10 and radio communication system
Wireless points communication between system or network.For example, wireless communication unit can include broadcast generation module, mobile communication module,
At least one in wireless Internet module, short range communication module and location message module.
Broadcast generation module and receive broadcast singal and/or broadcast correlation from external broadcast management server via broadcast channel
Message.Broadcast channel can include satellite channel and/or terrestrial channel.Broadcast management server can generate and send broadcast
Signal and/or broadcast related news server or reception before the broadcast singal that generates and/or broadcast related news and
Send it to the server of terminal.Broadcast singal can include TV broadcast singals, radio signals, data broadcasting signal
Etc..Moreover, broadcast singal may further include the broadcast singal combined with TV or radio signals.Broadcast correlation disappears
Breath can also be provided via mobile communications network, and in this case, broadcast related news can by mobile communication module Lai
Receive.Broadcast singal can exist in a variety of manners, for example, it can be referred to the electronic programming of DMB (DMB)
The form of southern (EPG), digital video or audio broadcasting hand-held (DVB-H) electronic service guidebooks (ESG) etc. and exist.Broadcast
Generation module can receive signal broadcast by using various types of broadcast systems.Especially, broadcast generation module can lead to
Cross wide using such as multimedia broadcasting-ground (DMB-T), DMB-satellite (DMB-S), digital video or audio
Broadcast-hand-held (DVB-H), forward link media (MediaFLO@) Radio Data System, received terrestrial digital broadcasting integrated service
Etc. (ISDB-T) digit broadcasting system receives digital broadcasting.Broadcast generation module, which may be constructed such that, is adapted to provide for broadcast letter
Number various broadcast systems and above-mentioned digit broadcasting system.The broadcast singal received via broadcast generation module and/or broadcast
Related news can be stored in memory 20 (or other types of storage medium).
Mobile communication module send radio signals to base station (for example, access point, node B etc.), exterior terminal with
And in server at least one and/or receive from it radio signal.Such radio signal can include language call
Signal, video or voice-frequency telephony signal or sent and/or received according to text and/or Multimedia Message it is various types of
Data.
Wireless Internet module supports the Wi-Fi (Wireless Internet Access) of task scheduling server.The module can be internal or external
Ground is couple to terminal.Wi-Fi (Wireless Internet Access) technology involved by the module can include WLAN (WLAN) (Wi-Fi),
Wibro (WiMAX), Wimax (worldwide interoperability for microwave accesses), HSDPA (high-speed downlink packet access) etc..
Short range communication module is the module for supporting junction service.Some examples of short-range communication technology include bluetoothTM、
Radio frequency identification (RFID), Infrared Data Association (IrDA), ultra wide band (UWB), purple honeybeeTMEtc..
Location message module is the module for checking or obtaining the location message of task scheduling server.Location message mould
The typical case of block is GPS (global positioning system).According to current technology, GPS module, which is calculated, comes from three or more satellites
Range message and correct time message and for the messages application triangulation of calculating so that according to longitude, latitude
Highly accurately calculate three-dimensional current location message.Currently, defended for calculating the method for position and time message using three
Star and the position calculated by using other satellite correction and the error of time message.In addition, GPS module can
By Continuous plus current location message in real time come calculating speed message.
Output unit is configured to provide output signal (for example, audio signal, regarding with vision, audio and/or tactile manner
Frequency or audio signal, alarm signal, vibration signal etc.).Output unit can include display unit, dio Output Modules, police
Declaration form member etc..
Display unit may be displayed on the message handled in task scheduling server 10.For example, when task scheduling server
10 when being in mode of operation, and display unit 51 can be shown for example, that text messaging, multimedia file are downloaded etc. is related
User interface (UI) or graphic user interface (GUI).When task scheduling server 10 be in video or voice-frequency telephony pattern or
During image capture mode, display unit 51 can show the image of capture and/or the image of reception, show video or audio or figure
UI or GUI of picture and correlation function etc..
Meanwhile, when display unit and touch pad in the form of layer it is superposed on one another to form touch-screen when, display unit can be with
As input unit and output device.Display unit can include liquid crystal display (LCD), thin film transistor (TFT) LCD (TFT-
LCD), at least one of Organic Light Emitting Diode (OLED) display, flexible display, three-dimensional (3D) display etc..This
Some in a little displays may be constructed such that transparence to allow user to be watched from outside, and this is properly termed as transparent display,
Typical transparent display can be, for example, TOLED (transparent organic light emitting diode) display etc..According to specific desired
Embodiment, task scheduling server 10 can include two or more display units (or other display devices), for example, task
Dispatch server can include outernal display unit (not shown) and inner display unit (not shown).Touch-screen can be used for detecting
Touch input pressure and touch input position and touch input area.
Dio Output Modules can be in logging mode, language identification mode, broadcast reception mould in task dispatch server
When under the isotypes such as formula, voice data transducing audio signal that is that wireless communication unit is received or storing in memory is simultaneously
And it is output as sound.Moreover, the specific function that dio Output Modules can be provided to task scheduling server 10 is performed is related
Audio output (for example, call signal receives sound, message sink sound etc.).Dio Output Modules can include loudspeaker,
Buzzer etc..
Alarm unit can provide output to notify event to task scheduling server 10.Typical event can
With including calling reception, message sink, key signals input, touch input etc..In addition to audio or video or audio output,
Alarm unit can provide output in a different manner with the generation of notification event.For example, alarm unit can be with the shape of vibration
Formula provides output, when receiving calling, message or some other entrance communications (incoming communication), alarm
Unit can provide tactile output (that is, vibrating) to notify to user.Exported by providing such tactile, even in
When the mobile phone at family is in the pocket of user, user also can recognize that the generation of various events.Alarm unit can also
The output of the generation of notification event is provided via display unit or dio Output Modules.
The order generation key input data that input block can be inputted according to user is with each of control task dispatch server
Plant operation.Input block allows user to input various types of message, and can include keyboard, metal dome, touch pad (example
Such as, the sensitive component of the change of resistance, pressure, electric capacity etc. caused by detection is due to being touched), roller, rocking bar etc..It is special
Not, when touch pad is superimposed on the display unit in the form of layer, touch-screen can be formed.In an embodiment of the present invention,
The input block includes touch screen and ink screen.
Interface unit is connected the interface that can pass through as at least one external device (ED) with task scheduling server 10.Example
Such as, external device (ED) can include wired or wireless head-band earphone port, external power source (or battery charger) port, You Xianhuo
Wireless data communications port, memory card port, the port for connecting the device with identification module, audio input/output (I/O) end
Mouth, video or audio I/O ports, ear port etc..Identification module can be that storage is used to verify that user uses task scheduling
The various message of server 10 and can include subscriber identification module (UIM), client identification module (SIM), Universal Subscriber know
Other module (USIM) etc..In addition, the device (hereinafter referred to as " identifying device ") with identification module can take smart card
Form, therefore, identifying device can be connected via port or other attachment means with task scheduling server 10.Interface unit can
For receiving the input (for example, data-message, electric power etc.) from external device (ED) and being transferred to the input received
One or more elements in task scheduling server 10 can be used between task dispatch server and external device (ED)
Transmit data.
In addition, when task scheduling server 10 is connected with external base, interface unit may be used as allowing to incite somebody to action by it
Electric power provides to the path of task dispatch server 10 from base or may be used as allowing the various orders inputted from base to believe
The path of task scheduling server number is transferred to by it.The various command signals or electric power inputted from base may be used as being used for
Whether identification mission dispatch server is accurately fitted within the signal on base.
Memory can store software program of the processing performed by controller and control operation etc., or can be temporary transient
Ground stores oneself data (for example, telephone directory, message, still image, video or audio etc.) through exporting or will export.And
And, memory can store vibration and the number of audio signal of the various modes on being exported when touching and being applied to touch-screen
According to.
Memory can include the storage medium of at least one type, and the storage medium includes flash memory, hard disk, multimedia
Card, card-type memory (for example, SD or DX memories etc.), random access storage device (RAM), static random-access memory
(SRAM), read-only storage (ROM), Electrically Erasable Read Only Memory (EEPROM), programmable read only memory
(PROM), magnetic storage, disk, CD etc..Moreover, task scheduling server 10 can be deposited with being performed by network connection
The network storage device cooperation of the store function of reservoir.
The overall operation of the usual control task dispatch server of controller.For example, controller is performed and language call, data
Communication, video or voice-frequency telephony etc. related control and processing.In addition, controller can be including more for reproducing (or playback)
The multi-media module of media data, multi-media module can be constructed in controller, or it is so structured that is separated with controller.
The handwriting input performed on the touchscreen or picture can be drawn input and are identified as by controller with execution pattern identifying processing
Character or image.
Power subsystem receives external power or internal power and provides each element of operation and group under the control of the controller
Appropriate electric power needed for part.
Various embodiments described herein can be with use such as computer software, hardware or its any combination of calculating
Machine computer-readable recording medium is implemented.Implement for hardware, embodiment described herein can be by using application-specific IC
(ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), scene can
Programming gate array (FPGA), processor, controller, microcontroller, microprocessor, it is designed to perform function described herein
At least one of electronic unit is implemented, and in some cases, such embodiment can be implemented in controller 180.
For software implementation, the embodiment of such as process or function can be with allowing to perform the single of at least one function or operation
Software module is implemented.Software code can by the software application (or program) write with any appropriate programming language Lai
Implement, software code can be stored in memory 160 and be performed by controller 180.
So far, oneself according to its function through describing task scheduling server.Below, for the sake of brevity, it will describe such as
Cunning in various types of task scheduling servers of folded form, board-type, oscillating-type, sliding-type task scheduling server etc.
Ejector half task scheduling server is used as example.Therefore, the present invention can be applied to any kind of task scheduling server, and
It is not limited to sliding-type task scheduling server.
Based on above-mentioned task scheduling server hardware structure, each embodiment of the inventive method is proposed.
Referring to Fig. 3, Fig. 3 is the functional block diagram of the task scheduling server of the embodiment of the present invention one.Task shown in Fig. 3
Dispatch server 10 includes judge module 101, acquisition module 102, processing module 103.Functional module is carried out specifically below
It is bright.
The judge module 101 of task scheduling server 10 judges whether current subtask is individually held by corresponding subsystem
OK.Acquisition module 102 is used for when current subtask is individually performed by corresponding subsystem, obtains the corresponding task in subtask
Script.Processing module 103 performs mission script, and the execution request of subtask is sent into corresponding subsystem, and monitors son and is appointed
The implementing result of business, determines whether subtask performs completion according to implementing result.
Supplementary notes, when confirming that current subtask runs succeeded, perform next subtask.Task scheduling is serviced
Before the judge module 101 of device 10 judges whether current subtask is individually performed by corresponding subsystem, task scheduling server
Safeguard that DAG schemes, preserve after DAG figures, DAG figures are converted into dag.py script files, the dag.py script files are in accordance with Airflow
The form generation of definition, dag.py scripts are submitted to Airflow workbenches by task scheduling server, and call Airflow
Platform interface, starts the task scheduling of dag.py scripts, and periodically performs dag.py script files, to obtain current son
Task, wherein, if subtask is Hive SQL tasks, the subtask is the subtask that subsystem is individually performed.
The task scheduling server that the present embodiment is provided can judge whether current subtask is individually held by corresponding subsystem
OK, when current subtask is individually performed by corresponding subsystem, the corresponding mission script in subtask is obtained, task pin is performed
This, corresponding subsystem is sent to by the execution request of subtask, and monitors the implementing result of subtask, true according to implementing result
Whether stator task performs completion, so, can determine that whether subtask runs succeeded, it is ensured that all subtasks can be performed in order,
Improve the efficiency of task scheduling.
Referring to Fig. 4, Fig. 4 is the functional block diagram of the task scheduling server of the embodiment of the present invention two.Task shown in Fig. 4
Dispatch server 10 includes judge module 101, acquisition module 102, processing module 103, wherein, processing module 103 includes performing
Module 104, monitoring modular 105.Functional module is described in detail below.
The judge module 101 of task scheduling server 10 judges whether current subtask is individually held by corresponding subsystem
OK.Acquisition module 102 is used for when current subtask is individually performed by corresponding subsystem, obtains the corresponding task in subtask
Script.Processing module 103 performs mission script, and the execution request of subtask is sent into corresponding subsystem, and monitors son and is appointed
The implementing result of business, determines whether subtask performs completion according to implementing result.
Supplementary notes, mission script includes first task node, the second task node, wherein, first task node
Definition performs request, and performing request is used to ask subtask system to perform current subtask, and the definition monitoring of the second task node please
Ask, monitoring request is used to monitor the implementing result that subtask system performs current subtask.Wherein, performing request includes hypertext
Host-host protocol is asked, and monitoring request includes polling request, and implementing result, which includes performing, to be completed or executive termination or execution
Cancel or perform exception.Please refer to Fig. 9, Fig. 9 is the code schematic diagram of the task node of the embodiment of the present invention 7 second,
Wherein, the code of the second task node includes PythonOperator classes, and sub-system execution pair is realized by Python scripts
Answer the state of subtask to be polled, the implementing result of subtask is obtained in time.
Specifically, performing module 104 is sent to subtask system according to first task node and performs request, and request is appointed
Business system performs current subtask.Monitoring modular 105 sends monitoring request, monitoring according to the second task node to subtask system
Subtask system performs the implementing result of current subtask, if implementing result is to run succeeded, it is determined that subtask runs succeeded,
If implementing result is executive termination or performs cancellation or perform exception, it is determined that subtask performs failure.Please join simultaneously
Fig. 9 is read, Fig. 9 is the code schematic diagram of the task node of the embodiment of the present invention 7 second, wherein, the code of the second task node includes
PythonOperator classes, the state for realizing sub-system execution correspondence subtask by Python scripts is polled, in time
Obtain the implementing result of subtask.
Supplementary notes, performing module 104 perform mission script first task node, according to first task node to
Subtask system sends hypertext transfer protocol requests.Monitoring modular 105 performs the second task node of mission script, according to the
Two task nodes send polling request to subtask system, and periodic monitor subtask system performs the execution knot of current subtask
Really, if implementing result is to run succeeded, it is determined that subtask runs succeeded, taken if implementing result is executive termination or execution
Disappear or perform exception, it is determined that subtask performs failure, if implementing result is runs succeeded, executive termination or perform takes
When disappearing or performing exception, then stop sending polling request, if implementing result is space-time, interval preset time sends poll
Request.
The task scheduling server that the present embodiment is provided can judge whether current subtask is individually held by corresponding subsystem
OK, when current subtask is individually performed by corresponding subsystem, the corresponding mission script in subtask is obtained, task pin is performed
This, corresponding subsystem is sent to by the execution request of subtask, and monitors the implementing result of subtask, if implementing result is to hold
Go successfully, it is determined that the first subtask runs succeeded, if implementing result is executive termination or performs cancellation or perform different
Often, it is determined that the first subtask performs failure, so, it can determine that whether subtask runs succeeded, it is ensured that all subtasks can have
Sequence is performed, and improves the efficiency of task scheduling.
Referring to Fig. 5, Fig. 5 is the application environment schematic diagram of the task scheduling server of the embodiment of the present invention three.Appointing in Fig. 5
Business dispatch server 10 includes judge module 101, acquisition module 102, processing module 103, wherein, processing module 103 includes holding
Row module 104, monitoring modular 105, task scheduling server 10 respectively with the first subsystem 501, the second subsystem the 502, the 3rd
Subsystem 503 is communicatively coupled.Fig. 5 is described in detail below.
The judge module 101 of task scheduling server 10 judges whether current subtask is individually held by corresponding subsystem
OK.Acquisition module 102 is used for when current subtask is individually performed by corresponding subsystem, obtains the corresponding task in subtask
Script.Processing module 103 performs mission script, and the execution request of subtask is sent into corresponding subsystem, and monitors son and is appointed
The implementing result of business, determines whether subtask performs completion according to implementing result.
For Fig. 5, the first subtask is that the first subsystem 501 is individually performed, the second subtask is the second subsystem
502 individually perform, the 3rd subtask is that the 3rd subsystem 503 is individually performed, it is assumed that current subtask is the first subtask
When, the judge module 101 of task scheduling server 10 judges that the first current subtask is individually held by corresponding first subsystem
OK.When current first subtask is individually performed by corresponding first subsystem, acquisition module 102 is used to obtain the first son times
It is engaged in corresponding mission script.Processing module 103 performs mission script, and the execution request of subtask is sent into corresponding first son
System, and the implementing result of the first subtask is monitored, determine whether the first subtask holds according to the implementing result of the first subtask
Row is completed.Supplementary notes, mission script includes first task node, the second task node, wherein, first task node is determined
Justice performs request, performs and asks to be used to ask subtask system to perform current subtask, the definition monitoring of the second task node is asked,
Monitoring request is used to monitor the implementing result that subtask system performs current subtask.Wherein, performing request includes hypertext biography
Defeated agreement request, monitoring request includes polling request, and implementing result includes performing completion or executive termination or performing taking
Disappear or perform exception.Please refer to Fig. 9, Fig. 9 is the code schematic diagram of the task node of the embodiment of the present invention 7 second, its
In, the code of the second task node includes PythonOperator classes, realizes that sub-system performs correspondence by Python scripts
The state of subtask is polled, and the implementing result of subtask is obtained in time.
Specifically, subtask can have multiple, for example, the first subtask, the second subtask, the 3rd subtask, currently
Subtask can be multiple subtasks in any one, when current subtask be the first subtask when, performing module 104
Sent according to first task node to the first subtask system and perform request, the first subtask of request system performs current first son
Task.Monitoring modular 105 sends monitoring request according to the second task node to the first subtask system, monitors the first subtask system
System performs the implementing result of current first subtask, if implementing result is to run succeeded, it is determined that subtask runs succeeded, if holding
Row result is executive termination or performs cancellation or perform exception, it is determined that subtask performs failure.
Supplementary notes, when current subtask is the first subtask, performing module 104 performs the of mission script
One task node, hypertext transfer protocol requests are sent according to first task node to the first subtask system.Monitoring modular 105
The second task node of mission script is performed, polling request, timing are sent to the first subtask system according to the second task node
The implementing result that subtask system performs current first subtask is monitored, if implementing result is to run succeeded, it is determined that the first son
Tasks carrying success, if implementing result is executive termination or performs cancellation or perform exception, it is determined that the first subtask
Failure is performed, if implementing result is runs succeeded, executive termination or perform when cancelling or performing abnormal, stops sending
Polling request, if implementing result is space-time, interval preset time sends polling request.
If the first subtask and the second subtask are serial execution relation, when current subtask is that the first subtask is held
When after going successfully, the subtask of acquisition second of task scheduling server 10, the judge module 101 of task scheduling server 10 judges
Whether the second subtask is individually performed by corresponding subsystem.Acquisition module 102 is used for when the second subtask is by corresponding the
When two subsystems are individually performed, the corresponding mission script in subtask is obtained.Processing module 103 performs mission script, by subtask
Execution request be sent to corresponding second subsystem, and monitor the implementing result of subtask, determine that son is appointed according to implementing result
Whether business performs completion.
Performing module 104 according to first task node to the and subtask system sends and performs request, request the and sub
Business system performs current second subtask.Monitoring modular 105 sends to the second subtask system according to the second task node and monitored
Request, the first subtask of monitoring system performs the implementing result of current second subtask, if implementing result is runs succeeded, really
Stator tasks carrying success, if implementing result is executive termination or performs cancellation or perform exception, it is determined that subtask
Perform failure.
Supplementary notes, when current subtask is the second subtask, performing module 104 performs the of mission script
One task node, hypertext transfer protocol requests are sent according to first task node to the second subtask system.Monitoring modular 105
The second task node of mission script is performed, polling request, timing are sent to the second subtask system according to the second task node
The implementing result that subtask system performs current second subtask is monitored, if implementing result is to run succeeded, it is determined that the first son
Tasks carrying success, if implementing result is executive termination or performs cancellation or perform exception, it is determined that the second subtask
Failure is performed, if implementing result is runs succeeded, executive termination or perform when cancelling or performing abnormal, stops sending
Polling request, if implementing result is space-time, interval preset time sends polling request.If the first subtask and the second subtask
For parallel execution relation, and when performing three subtasks again after having performed the first subtask and the second subtask, task is adjusted
Degree server 10 performs the 3rd subtask again after the first subtask and the second subtask has been performed, and obtains the 3rd subtask, to
The request of 3rd subsystem performs the 3rd subtask, and monitors whether the 3rd subtask runs succeeded.
The task scheduling server that the present embodiment is provided can judge whether current subtask is individually held by corresponding subsystem
OK, when current subtask is individually performed by corresponding subsystem, the corresponding mission script in subtask is obtained, task pin is performed
This, corresponding subsystem is sent to by the execution request of subtask, and monitors the implementing result of subtask, true according to implementing result
Whether stator task performs completion, if implementing result is to run succeeded, it is determined that the first subtask runs succeeded, if implementing result
It is that exception is cancelled or performed in executive termination or execution, it is determined that the first subtask performs failure, if implementing result is to hold
Go successfully, executive termination or perform when cancelling or performing abnormal, then stop sending polling request, if implementing result is sky
When, then it is spaced preset time and sends polling request, so, can determine that whether subtask runs succeeded, it is ensured that all subtask energy
Perform in order, improve the efficiency of task scheduling.
The present invention also provides a kind of method for scheduling task, and this method is applied to the task scheduling server 10 shown in Fig. 3-5,
The method for scheduling task of the present embodiment is described in detail below.
Refering to Fig. 6, Fig. 6 is the flow chart of the method for scheduling task of the embodiment of the present invention four.
In step s 601, whether the judge module 101 of task scheduling server 10 judges current subtask by corresponding
Subsystem is individually performed.
In step S602, acquisition module 102 is used for when current subtask is individually performed by corresponding subsystem, obtains
Take the corresponding mission script in subtask.
In step S603, processing module 103 performs mission script, and the execution request of subtask is sent into corresponding son
System, and the implementing result of subtask is monitored, determine whether subtask performs completion according to implementing result.
Supplementary notes, mission script includes first task node, the second task node, wherein, first task node
Definition performs request, and performing request is used to ask subtask system to perform current subtask, and the definition monitoring of the second task node please
Ask, monitoring request is used to monitor the implementing result that subtask system performs current subtask.Performing request includes Hyper text transfer
Agreement request, monitoring request includes polling request, implementing result includes performing completion or executive termination or perform cancellation,
Or perform exception.Please refer to Fig. 9, Fig. 9 is the code schematic diagram of the task node of the embodiment of the present invention 7 second, wherein,
The code of second task node includes PythonOperator classes, realizes that sub-system performs correspondence and appointed by Python scripts
The state of business is polled, and the implementing result of subtask is obtained in time.
Further remark additionally, the judge module 101 of task scheduling server 10 judge current subtask whether by
Before corresponding subsystem is individually performed, task scheduling server safeguards that DAG schemes, and preserves after DAG figures, DAG figures are converted into
Dag.py script files, the dag.py script files are generated in accordance with the Airflow forms defined, and task scheduling server will
Dag.py scripts are submitted to Airflow workbenches, and call Airflow platform interfaces, and the starting dag.py scripts of the task is adjusted
Degree, and dag.py script files are periodically performed, to obtain current subtask.When the current subtask of confirmation runs succeeded
When, next subtask is performed, wherein, if subtask is Hive SQL tasks, the subtask is what subsystem was individually performed
Subtask.
The method for scheduling task that the present embodiment is provided can judge whether current subtask is individually performed by corresponding subsystem,
When current subtask is individually performed by corresponding subsystem, the corresponding mission script in subtask is obtained, mission script is performed,
The execution request of subtask is sent to corresponding subsystem, and monitors the implementing result of subtask, is determined according to implementing result
Whether subtask performs completion, so, can determine that whether subtask runs succeeded, it is ensured that all subtasks can be performed in order, be carried
The efficiency of high task scheduling.
Refering to Fig. 7, Fig. 7 is the flow chart of the method for scheduling task of the embodiment of the present invention five.This method is applied to shown in Fig. 3-5
Task scheduling server 10, the method for scheduling task of the present embodiment is described in detail below.
In step s 701, performing module 104 sends to subtask system according to first task node and performs request, asks
Subtask system performs current subtask.
In step S702, monitoring modular 105 sends monitoring request, monitoring according to the second task node to subtask system
Subtask system performs the implementing result of current subtask.Supplementary notes, the code of the second task node includes
PythonOperator classes, the state for realizing sub-system execution correspondence subtask by Python scripts is polled, in time
Obtain the implementing result of subtask.
In step S703, if implementing result is to run succeeded, it is determined that subtask runs succeeded, if implementing result is to hold
Row, which is terminated or performed, cancels or performs exception, it is determined that subtask performs failure.
The method for scheduling task that the present embodiment is provided can monitor the implementing result of subtask, if implementing result is to perform into
Work(, it is determined that subtask runs succeeded, if implementing result is executive termination or performs cancellation or perform exception, it is determined that
Subtask performs failure, so, can determine that whether subtask runs succeeded, it is ensured that all subtasks can be performed in order, improves and appoints
The efficiency of business scheduling.
Refering to Fig. 8, Fig. 8 is the flow chart of the method for scheduling task of the embodiment of the present invention six.This method is applied to shown in Fig. 3-5
Task scheduling server 10, the method for scheduling task of the present embodiment is described in detail below.
In step S801, performing module 104 perform mission script first task node, according to first task node to
Subtask system sends hypertext transfer protocol requests.
In step S802, monitoring modular 105 perform mission script the second task node, according to the second task node to
Subtask system sends polling request.Wherein, the code of the second task node includes PythonOperator classes, passes through Python
Script realizes that the state of sub-system execution correspondence subtask is polled, and the implementing result of subtask is obtained in time.
In step S803, periodic monitor subtask system performs the implementing result of current subtask, if implementing result is
Run succeeded, it is determined that subtask runs succeeded, if implementing result is executive termination or performs cancellation or perform exception,
Then determine that subtask performs failure.
In step S804, if implementing result is runs succeeded, exception is cancelled or performed in executive termination or execution
When, then stop sending polling request, if implementing result is space-time, interval preset time sends polling request.
The method for scheduling task that the present embodiment is provided can monitor the implementing result of subtask, if implementing result is to perform into
When work(, executive termination or execution are cancelled or perform abnormal, then stop transmission polling request, if implementing result is space-time,
Then interval preset time sends polling request, so, can determine that whether subtask runs succeeded, it is ensured that all subtasks can be in order
Perform, improve the efficiency of task scheduling.
These are only presently preferred embodiments of the present invention, be not intended to limit the invention, it is all the present invention spirit and
Any modifications, equivalent substitutions and improvements made within principle etc., should be included in the scope of the protection.
Claims (10)
1. a kind of task scheduling server, it is characterised in that including：
Judge module, for judging whether current subtask is individually performed by corresponding subsystem；
Acquisition module, for when current subtask is individually performed by corresponding subsystem, obtaining the subtask corresponding
Mission script；
Processing module, for performing the mission script, the corresponding subsystem is sent to by the execution request of the subtask
System, and the implementing result of the subtask is monitored, determine whether the subtask performs completion according to the implementing result.
2. task scheduling server as claimed in claim 1, it is characterised in that the mission script includes first task section
Point, the second task node, wherein, the first task node definition performs request, described to perform request for asking the son
Task system performs the current subtask, and the second task node definition monitoring request, the monitoring request is used to monitor
The subtask system performs the implementing result of the current subtask.
3. task scheduling server as claimed in claim 2, it is characterised in that the execution request includes Hyper text transfer association
View is asked, and the monitoring request includes polling request, and the implementing result includes performing completion or executive termination or held
Exception is cancelled or performed to row.
4. task scheduling server as claimed in claim 2 or claim 3, it is characterised in that the processing module includes：
Performing module, for performing request to subtask system transmission is described according to the first task node, asks institute
State subtask system and perform the current subtask；
Monitoring modular, asks for sending the monitoring to the subtask system according to second task node, monitors institute
The implementing result that subtask system performs the current subtask is stated, if the implementing result is to run succeeded, it is determined that described
Subtask runs succeeded, if the implementing result is executive termination or performs cancellation or perform exception, it is determined that described
Subtask performs failure.
5. task scheduling server as claimed in claim 3, it is characterised in that the performing module, is additionally operable to execution described
The first task node of mission script, Hyper text transfer is sent according to the first task node to the subtask system
Agreement request；
The monitoring modular, is additionally operable to perform second task node of the mission script according to second task node
Polling request is sent to the subtask system, subtask system described in periodic monitor performs the execution knot of the current subtask
Really, if the implementing result is to run succeeded, it is determined that the subtask runs succeeded, if the implementing result is to perform end
Only or perform and cancel or performs exception, it is determined that the subtask performs failure, if the implementing result is to perform into
When work(, executive termination or execution are cancelled or perform abnormal, then stop the transmission polling request, if the implementing result
For space-time, then it is spaced preset time and sends the polling request.
6. a kind of method for scheduling task, it is characterised in that including：
Judge whether current subtask is individually performed by corresponding subsystem；
When current subtask is individually performed by corresponding subsystem, the corresponding mission script in the subtask is obtained；
The mission script is performed, the execution request of the subtask the corresponding subsystem is sent to, and monitor described
The implementing result of subtask, determines whether the subtask performs completion according to the implementing result.
7. method for scheduling task as claimed in claim 6, it is characterised in that the mission script include first task node,
Second task node, wherein, the first task node definition performs request, described to perform request for asking the subtask
System performs the current subtask, and second task node defines monitoring request, and the monitoring request is described for monitoring
Subtask system performs the implementing result of the current subtask.
8. method for scheduling task as claimed in claim 7, it is characterised in that the execution request includes HTTP
Request, the monitoring request includes polling request, and the implementing result, which includes performing, to be completed or executive termination or execution
Cancel or perform exception.
9. method for scheduling task as claimed in claim 7 or 8, it is characterised in that perform the mission script, the son is appointed
The execution request of business is sent to the corresponding subsystem, and monitors the implementing result of the subtask, and knot is performed according to described
Fruit determines whether the subtask performs completion, including：
Request is performed to subtask system transmission is described according to second task node, asks the subtask system to be held
The row current subtask；
The monitoring is sent to the subtask system to ask, monitor the subtask system and hold according to second task node
The implementing result of the row current subtask；
If the implementing result is to run succeeded, it is determined that the subtask runs succeeded；
If the implementing result is executive termination or performs cancellation or perform exception, it is determined that the subtask performs mistake
Lose.
10. method for scheduling task as claimed in claim 8, it is characterised in that the execution mission script, by the son
The execution request of task is sent to the corresponding subsystem, and monitors the implementing result of the subtask, is performed according to described
As a result determine whether the subtask performs completion, including：
The first task node of the mission script, second task node are performed, according to the first task node
Hypertext transfer protocol requests are sent to the subtask system；
Polling request is sent to the subtask system according to second task node, subtask system is held described in periodic monitor
The implementing result of the row current subtask；
If the implementing result is to run succeeded, it is determined that the subtask runs succeeded；
If the implementing result is executive termination or performs cancellation or perform exception, it is determined that the subtask performs mistake
Lose；
If the implementing result is runs succeeded, executive termination or perform when cancelling or performing abnormal, stop sending
The polling request；
If the implementing result is space-time, interval preset time sends the polling request.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710287419.3A CN107133089A (en) | 2017-04-27 | 2017-04-27 | A kind of task scheduling server and method for scheduling task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710287419.3A CN107133089A (en) | 2017-04-27 | 2017-04-27 | A kind of task scheduling server and method for scheduling task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107133089A true CN107133089A (en) | 2017-09-05 |
Family
ID=59715259
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710287419.3A Pending CN107133089A (en) | 2017-04-27 | 2017-04-27 | A kind of task scheduling server and method for scheduling task |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107133089A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107608774A (en) * | 2017-09-08 | 2018-01-19 | 新智云数据服务有限公司 | A kind of method for scheduling task, device, equipment and storage medium |
| CN108829505A (en) * | 2018-06-28 | 2018-11-16 | 北京奇虎科技有限公司 | A kind of distributed scheduling system and method |
| CN109120706A (en) * | 2018-08-28 | 2019-01-01 | 中国邮政储蓄银行股份有限公司 | Business scheduling method and system |
| CN109213754A (en) * | 2018-03-29 | 2019-01-15 | 北京九章云极科技有限公司 | A kind of data processing system and data processing method |
| CN109840267A (en) * | 2019-03-01 | 2019-06-04 | 成都品果科技有限公司 | A kind of ETL process system and method |
| CN110609734A (en) * | 2018-06-15 | 2019-12-24 | 北京京东尚科信息技术有限公司 | Method and device for scheduling timed tasks |
| CN110764747A (en) * | 2019-10-22 | 2020-02-07 | 南方电网科学研究院有限责任公司 | Data calculation scheduling method based on Airflow |
| CN113674797A (en) * | 2020-05-15 | 2021-11-19 | 复旦大学 | Proteome-based data detection system, method, device and storage medium |
-
2017
- 2017-04-27 CN CN201710287419.3A patent/CN107133089A/en active Pending
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107608774A (en) * | 2017-09-08 | 2018-01-19 | 新智云数据服务有限公司 | A kind of method for scheduling task, device, equipment and storage medium |
| CN109213754A (en) * | 2018-03-29 | 2019-01-15 | 北京九章云极科技有限公司 | A kind of data processing system and data processing method |
| CN109213754B (en) * | 2018-03-29 | 2020-02-28 | 北京九章云极科技有限公司 | Data processing system and data processing method |
| CN110609734A (en) * | 2018-06-15 | 2019-12-24 | 北京京东尚科信息技术有限公司 | Method and device for scheduling timed tasks |
| CN108829505A (en) * | 2018-06-28 | 2018-11-16 | 北京奇虎科技有限公司 | A kind of distributed scheduling system and method |
| CN109120706A (en) * | 2018-08-28 | 2019-01-01 | 中国邮政储蓄银行股份有限公司 | Business scheduling method and system |
| CN109840267A (en) * | 2019-03-01 | 2019-06-04 | 成都品果科技有限公司 | A kind of ETL process system and method |
| CN109840267B (en) * | 2019-03-01 | 2023-04-21 | 成都品果科技有限公司 | Data ETL system and method |
| CN110764747A (en) * | 2019-10-22 | 2020-02-07 | 南方电网科学研究院有限责任公司 | Data calculation scheduling method based on Airflow |
| CN113674797A (en) * | 2020-05-15 | 2021-11-19 | 复旦大学 | Proteome-based data detection system, method, device and storage medium |
| CN113674797B (en) * | 2020-05-15 | 2024-05-14 | 复旦大学 | Proteome-based data detection system, method, apparatus and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107133089A (en) | A kind of task scheduling server and method for scheduling task | |
| CN106302385B (en) | A kind of message distribution device and method | |
| CN104883658B (en) | The processing method of virtual card information and system | |
| CN105100603B (en) | A kind of triggering camera arrangement and its method that intelligent terminal is embedded | |
| CN105208082B (en) | A kind of method and device for instructing user's using terminal, terminal | |
| US20110244928A1 (en) | Hands-free helper | |
| CN106453372B (en) | A kind of terminal, server and account logon method | |
| CN104917879B (en) | A kind of method, device and mobile terminal for cancelling alarm clock calling | |
| CN105718568B (en) | A kind of device and method of the unified interaction of mostly application | |
| CN106941443A (en) | One population historical record checks terminal and method | |
| CN106383780A (en) | Monkey test method and device | |
| CN102427430A (en) | Two-way communication of events between a mobile device and remote client | |
| CN105099875A (en) | Method and device for multiple users to collaboratively edit and publish picture information | |
| CN105278995B (en) | Management method, system, server and the mobile terminal of application program | |
| CN107104970A (en) | A kind of regular and synchronized processing method, system, rule server and gray scale server | |
| CN105100477B (en) | Terminal notifying device and method | |
| CN106550124A (en) | It is a kind of to apply prepackage method and terminal | |
| CN105141738A (en) | Volume adjusting method and device | |
| CN107341013A (en) | One kind start fault handling method, equipment and computer-readable recording medium | |
| CN106385333A (en) | Automatic deploying method and system for application program | |
| CN106598538A (en) | Method and system for updating instruction set | |
| CN114510305B (en) | Model training method and device, storage medium and electronic equipment | |
| CN113051094A (en) | Supervision data submission testing method and device | |
| CN112256372A (en) | Information processing method and device and electronic equipment | |
| CN106878093A (en) | One kind is without response log analytic method and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170905 |
|
| RJ01 | Rejection of invention patent application after publication |