CN108228328A - A kind of stream task implementation method, device and electronic equipment - Google Patents
A kind of stream task implementation method, device and electronic equipment Download PDFInfo
- Publication number
- CN108228328A CN108228328A CN201810010283.6A CN201810010283A CN108228328A CN 108228328 A CN108228328 A CN 108228328A CN 201810010283 A CN201810010283 A CN 201810010283A CN 108228328 A CN108228328 A CN 108228328A
- Authority
- CN
- China
- Prior art keywords
- task
- stream
- master
- streams
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Hardware Redundancy (AREA)
Abstract
Task implementation method, device and electronic equipment are flowed an embodiment of the present invention provides a kind of, the method includes：A stream task is selected to flow task as master in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms, and using other stream tasks in the multiple stream task in addition to the master streams task as follower stream tasks；The corresponding computer room of the master streams task is controlled to perform the master streams task；When detecting that the master streams task occurs abnormal in the process of implementation, a stream task is selected in the follower streams task as current master and flows task；And the corresponding computer room of the current master streams task is controlled to perform the current master streams task.The embodiment of the present invention when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of stream task implementation method, device and electronic equipment.
Background technology
Distributed real-time streams task framework, such as Spark Streaming, Flink, Storm etc. can be achieved on a large scale
Real-time streaming data processing.When realizing real-time streaming data processing, delay machine when mistake occur in certain machine or a few machines
Afterwards, the task on the machine for mistake occur can be rescheduled on normal machine and run by distributed real-time streams task framework,
To ensure the normal operation of stream task.
However, inventor has found in the implementation of the present invention, at least there are the following problems for the prior art：Existing point
Cloth real-time streams task framework can not handle the not available situation of entire cluster of operation stream task, such as：1st, stream task is run
Cluster there is exception or need maintenance shut-downs；2. mistake causes the cluster for running stream task unavailable to stream task frame in itself；
3. network of computer room exception where running the cluster of stream task etc..When there are these problems in the cluster for running stream task, distribution
Formula real-time streams task framework is disabled.
Invention content
The embodiment of the present invention is designed to provide a kind of stream task implementation method, device and electronic equipment, to run
When the cluster of stream task breaks down, ensure the normal operation of stream task.Specific technical solution is as follows：
In order to achieve the above objectives, in a first aspect, the embodiment of the invention discloses a kind of stream task implementation method, this method packet
It includes：
A stream task is selected to be flowed as master in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Task, and other stream tasks in the multiple stream task in addition to the master streams task are appointed as follower streams
Business；
The corresponding computer room of the master streams task is controlled to perform the master streams task；
When detecting that the master streams task occurs abnormal in the process of implementation, in the follower streams task
A stream task is selected as current master and flows task；
The corresponding computer room of the current master streams task is controlled to perform the current master streams task.
Optionally, the above method further includes：
In the corresponding computer room of master stream tasks performs the master streams task process, the master is controlled
The corresponding computer room of stream task stores the execution state of master stream tasks to external data base according to prefixed time interval
Or file system；
The control current master flows the corresponding computer room of task and performs the current master streams task, including：
The corresponding computer room of the current master streams task is controlled from described in the external data base or file system reading
Master flows the execution state of task, and the state of the current master streams task is restored to the execution state, performs
The current master after state is restored flows task.
Optionally, the above method further includes：
In the current master flows the corresponding computer room execution of the task current master streams task process, institute is controlled
State the execution state that the current master is flowed task according to prefixed time interval by the corresponding computer room of current master streams task
It stores to external data base or file system.
Optionally, it is described that a stream task is selected in the follower streams task as current master stream tasks
After step, the above method further includes：
After detecting that master stream task recovery is normal, will restore the normal master stream tasks as
Follower flows task.
Optionally, a stream task is selected to make in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Task is flowed for master, including：
A stream task is selected in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms by Zookeeper
Task is flowed as master；
It is described selected in the follower streams task one stream task as current master flow task, including：
A stream task is selected to be flowed as current master in the follower streams task by the Zookeeper
Task.
Second aspect, the embodiment of the invention discloses a kind of stream task realization device, which includes：
First choice module, for one stream of selection in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Task flows task, and other stream tasks in the multiple stream task in addition to the master streams task are made as master
Task is flowed for follower；
First control module, for the corresponding computer room of the master streams task to be controlled to perform the master streams task；
Second selecting module, for when detecting that master stream task occurs abnormal in the process of implementation, in institute
It states and a stream task is selected to flow task as current master in follower stream tasks；
Second control module, for the corresponding computer room of the current master streams task to be controlled to perform the current master
Stream task.
Optionally, above device further includes：
Third control module, for performing the master streams task process in the corresponding computer room of master stream tasks
In, control the master streams task corresponding computer room that the master is flowed to the execution state of task according to prefixed time interval
It stores to external data base or file system；
Second control module, specifically for controlling the corresponding computer room of the current master streams task from the outside
Database or file system read the execution state of the master streams task, and the current master is flowed to the state of task
Restore to the execution state, perform the current master streams task after state is restored.
Optionally, above device further includes：
4th control module performs the current master streams for flowing the corresponding computer room of task in the current master
In task process, the corresponding computer room of the control current master stream task is according to prefixed time interval by the current master
The execution state of stream task is stored to external data base or file system.
Optionally, above device further includes：
Recovery module, for a stream task being selected to make in the follower streams task in second selecting module
After current master streams task, after detecting that the master streams task recovery is normal, it will restore normal described
Master flows task and flows task as follower.
Optionally, the first choice module, the logic disposed in advance in multiple computer rooms specifically for passing through Zookeeper
A stream task is selected in identical multiple stream tasks as master and flows task；
Second selecting module, specifically for being selected in the follower streams task by the Zookeeper
One stream task flows task as current master.
The third aspect, an embodiment of the present invention provides a kind of electronic equipment, including processor, communication interface, memory and
Communication bus, wherein, processor, communication interface, memory completes mutual communication by communication bus；
Memory, for storing computer program；
Processor during for performing the program stored on memory, realizes that as above the stream task described in first aspect is real
Existing method and step.
Fourth aspect, the embodiment of the present invention additionally provide a kind of computer readable storage medium, described computer-readable to deposit
Instruction is stored in storage media, when run on a computer so that computer performs the as above stream described in first aspect and appoints
Pragmatic existing method and step.
5th aspect, the embodiment of the present invention additionally provides a kind of computer program product for including instruction, when it is being calculated
When being run on machine so that computer performs the method and step that as above the stream task described in first aspect is realized.
It is provided in an embodiment of the present invention to flow task implementation method, device and electronic equipment, by the advance portion of multiple computer rooms
A stream task is selected in the identical multiple stream tasks of the logic of administration as master stream tasks, and will be in the multiple stream task
Other stream tasks in addition to the master streams task flow task as follower；The master streams task is controlled to correspond to
Computer room perform master stream task；When detecting that the master streams task occurs abnormal in the process of implementation,
A stream task is selected in the follower streams task as current master and flows task；The current master streams is controlled to appoint
Be engaged in corresponding computer room performs the current master streams task.The embodiment of the present invention event can occur in the cluster of operation stream task
During barrier, ensure the normal operation of stream task.
Certainly, it implements any of the products of the present invention or method must be not necessarily required to reach all the above excellent simultaneously
Point.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described.
Fig. 1 is the flow diagram of the first stream task implementation method provided in an embodiment of the present invention；
Fig. 2 is the flow diagram of second provided in an embodiment of the present invention stream task implementation method；
Fig. 3 is the flow diagram of the third stream task implementation method provided in an embodiment of the present invention；
Fig. 4 is the flow diagram of 4th kind provided in an embodiment of the present invention stream task implementation method；
Fig. 5 is the structure diagram of the first stream task realization device provided in an embodiment of the present invention；
Fig. 6 is the structure diagram of second provided in an embodiment of the present invention stream task realization device；
Fig. 7 is the structure diagram of the third stream task realization device provided in an embodiment of the present invention；
Fig. 8 is the structure diagram of 4th kind provided in an embodiment of the present invention stream task realization device；
Fig. 9 is the structure diagram of a kind of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes.
Fig. 1 be it is provided in an embodiment of the present invention the first stream task implementation method flow diagram, this method include with
Lower step：
S101, select in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms a stream task as
Master flows task, and other stream tasks in multiple stream tasks in addition to master stream tasks are appointed as follower streams
Business.
In distributed stream task framework, such as Spark Streaming, Flink, Storm, distributed stream task are drawn
Divide several tasks, accordingly perform corresponding task respectively by several machines in cluster.Several machines one in cluster
As deposit in a computer room.After certain machine in cluster goes wrong, distributed stream task framework can will go wrong
Machine to be performed or other in the task scheduling to cluster that is carrying out be on the machine of normal operating conditions and continue
It performs.
It, can be in the present embodiment for the situation that entire cluster breaks down in the implementation procedure of distributed stream task
In advance in the identical multiple stream tasks of multiple computer rooms deployment logic, i.e., dispose the identical stream task of logic in each computer room.It patrols
Identical stream task is collected to refer to：Stream task is divided into the division rule of several tasks and identical and different of division result
Data dependence relation between business is identical.Specifically, several machines execution pair in cluster that can be by depositing in each computer room
Answering for task.
For example, stream task A is divided into：Task a, task b, task c, then can in 3 computer rooms, i.e., computer room 1, computer room 2,
Computer room 3 disposes the identical stream task A of logic.Specifically, the machine 11,12,13 in the cluster 1 that computer room 1 is stored is held respectively successively
Row task a, b, c；Equally, the machine 21,22,23 in the cluster 2 that computer room 2 is stored performs task a, b, c respectively successively；Computer room 3
Machine 31,32,33 in the cluster 3 of storage performs task a, b, c respectively successively.
It should be noted that deploying the identical stream task of logic in multiple computer rooms, it is not meant in the same period
Interior each computer room is carried out stream task, in fact, only there are one computer rooms within the same period to perform stream task.Thus, it is flowing
A stream task is selected to hold before task is not opened, in the stream task for needing the logic disposed in advance in multiple computer rooms identical
Row, the stream task for being chosen to perform are properly termed as master stream tasks.
In a kind of realization method, one stream of selection is appointed in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Business flows task as master, can include：
A stream task is selected in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms by Zookeeper
Task is flowed as master.
Zookeeper is a kind of distributed application program coordination service, can provide Consistency service for Distributed Application,
The service of offer includes：Configuring maintenance, domain name service, distributed synchronization, group service etc..The core mechanism of general Zookeeper
Including：Select main flow and synchronous flow.When service just startup, leader collapses, follower deficiency halves, system is put into
Main flow is selected, service is not provided externally at this time；After leader is elected, system is put into synchronous flow, between service
Completion status synchronizes, and hereafter externally provides service.In main flow is selected, common algorithm is calculated for FashLeaderElection
Method.
In the present embodiment, stream task it is not actuated before, can the election mechanism based on Zookeeper, especially by
One stream of selection is appointed in the identical multiple stream tasks of logic that FashLeaderElection algorithms are disposed in advance in multiple computer rooms
Business as be chosen perform master stream task, be equivalent to the election mechanism based on Zookeeper, in multiple stream tasks into
Row for the first time lead by choosing.Above-mentioned FashLeaderElection algorithms belong to the prior art, and the present embodiment is not described in detail herein.
It is appreciated that after select one to flow task as master stream tasks in above-mentioned multiple stream tasks, then can be with
Other stream tasks in multiple stream tasks in addition to master stream tasks are flowed into task as follower.Follower flows task
It can participate in voting, but after master stream tasks are selected in main flow in selecting for Zookeeper, follower stream tasks
Specific task is not performed.
S102, the corresponding computer room of control master stream tasks perform master stream tasks.
After selecting master stream tasks, the corresponding computer room of master stream tasks can be controlled according to advance convection current task
It carries out the data dependence relation between the division result and different task of division task and performs master stream tasks.Specifically, may be used
It is performed with several machines in the cluster by depositing in the corresponding computer room of master stream tasks according to division result and execution sequence
Corresponding task.
For example, in the example of step S101, if the stream task A selected as master that computer room 1 is disposed flows task, that
, can task a, b, c be performed by the machine 11,12,13 in cluster 1 that computer room 1 is stored respectively successively.
Certainly, when the corresponding computer room of master stream tasks performs master stream tasks, follower stream tasks are corresponding
Computer room does not perform corresponding task, i.e. follower stream tasks are in standby states.In step S101 particular by
In the case that Zookeeper selects master stream tasks, master stream tasks are performed in the corresponding computer room of master stream tasks
During, follower streams task can be used as listener, monitor the execution state hair about stream task from Zookeeper
The raw information changed.That is, when the execution state for flowing task changes, execution that Zookeeper can will flow task
State change message notifies to flow task to follower.It is appreciated that the specifically, execution shape that Zookeeper can will flow task
State changes information and notifies computer room corresponding to follower stream tasks.
For example, in the example of step S101, if the stream task A selected as master that computer room 1 is disposed flows task, that
, the stream task A selected as follower that computer room 2, computer room 3 are disposed can be flowed task.
S103 when detecting that master stream tasks occur abnormal in the process of implementation, is selected in follower stream tasks
It selects a stream task and flows task as current master.
In the present embodiment, there is exception in the process of implementation in master stream tasks to be included：1st, the collection of stream task is run
There is exception or needs maintenance shut-downs in group；2. mistake causes the cluster for running stream task unavailable to stream task frame in itself；3. fortune
Network of computer room exception where the cluster of row stream task etc..
After master stream tasks exception occur in the process of implementation, in order to ensure the normal execution of stream task, Ke Yi
A stream task is selected in follower stream tasks in standby states as current master and flows task, with by current
The corresponding computer room of master stream tasks continues to execute stream task.
In a kind of realization method, a stream task is selected to appoint as current master streams in the follower streams task
Business, can include：
One stream task is selected as current master in follower stream tasks by Zookeeper and flows task.
In this realization method, the election mechanism based on Zookeeper is equivalent to, standby shapes are in currently available
Second of choosing master is carried out in the follower stream tasks of state.Concrete implementation process can refer to the first time choosing in step S101
Main realization process, which is not described herein again.
For example, in the example of step S101, the current master of stream task A selected as that computer room 2 is disposed flows task.
S104 controls the corresponding computer room of current master streams task to perform current master streams task.
After selecting current master streams task, the corresponding computer room of current master streams task can be controlled according to advance
The data dependence relation that convection current task is carried out between the division result and different task of division task performs current master streams and appoints
Business.Specifically, several machines in cluster that can be by depositing in the corresponding computer room of current master streams task are tied according to division
Fruit and execution sequence perform corresponding task.
For example, in the example of step S101, if the current master streams of the stream task A selected as that computer room 2 is disposed are appointed
Business, then, the cluster 1 that can be stored by computer room 2 performs stream task A.
Stream task implementation method provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Fig. 2 be second provided in an embodiment of the present invention stream task implementation method flow diagram, this method include with
Lower step：
S201, select in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms a stream task as
Master flows task, and other stream tasks in multiple stream tasks in addition to master stream tasks are appointed as follower streams
Business.
S202, the corresponding computer room of control master stream tasks perform master stream tasks.
In the present embodiment, above-mentioned steps S201-S202 can respectively with the step S101-S102 in embodiment illustrated in fig. 1
Identical, which is not described herein again.
S203, in the corresponding computer room of master stream tasks performs master stream task process, control master stream tasks
Corresponding computer room stores the execution state of master stream tasks to external data base or file system according to prefixed time interval.
The execution state of master stream tasks can include：Which current processed data of master stream tasks and
Which current untreated data etc..It can also include：Which task dispatching that master stream tasks have specifically been divided in execution.
Said external database can include：MySQL database etc..
In the corresponding computer room of master stream tasks performs master stream task process, master can be controlled to flow task pair
The computer room answered stores the execution state of master stream tasks to external data base or file system according to prefixed time interval, this
Sample when master stream tasks occur in the process of implementation such as exception in step S103, takes over any of master stream tasks
Follower stream tasks can know the current execution state of master stream tasks, and perform stream since the execution state and appoint
Business, without from the beginning performing entire stream task.Prefixed time interval can be set according to industry.
It is appreciated that step S203 is performed while step S202 is performed.
S204 when detecting that master stream tasks occur abnormal in the process of implementation, is selected in follower stream tasks
It selects a stream task and flows task as current master.
In the present embodiment, above-mentioned steps S204 can be identical with the step S103 in embodiment illustrated in fig. 1 respectively, this
In repeat no more.
S205 controls the corresponding computer room of current master streams task to perform current master streams task.
In the present embodiment, the corresponding computer room of current master streams task is controlled to perform current master streams task, can be wrapped
It includes：The corresponding computer room of current master streams task is controlled to read the execution of master stream tasks from external data base or file system
State, and the state of current master stream tasks is restored to execution state, it performs the current master streams after state is restored and appoints
Business.
For example, the stream task A that computer room 1 is disposed flows task for master, stream task A is divided into：Task a, task b, appoint
Be engaged in c, and in step S203, computer room 1 stores the execution state of master stream tasks to external data according to prefixed time interval
Library or file system.Assuming that it is the machine 11 in the cluster 1 that computer room 1 is stored that master, which flows the current transmission of task to perform state,
Execution task a is finished.Current master streams task is the stream task A that computer room 2 is disposed, then computer room 2 is from external data base or file
System reads the current execution state of master stream tasks, and the state of current master stream tasks is restored to execution state,
Currently the state of master streams task it will restore to finish this state to execution task a, after then computer room 2 performs state recovery
Current master stream task.Specifically, the machine 22 in the cluster 2 that computer room 2 is stored starts execution task b.
Stream task implementation method provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Fig. 3 be it is provided in an embodiment of the present invention the third stream task implementation method flow diagram, this method include with
Lower step：
S301, select in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms a stream task as
Master flows task, and other stream tasks in multiple stream tasks in addition to master stream tasks are appointed as follower streams
Business.
S302, the corresponding computer room of control master stream tasks perform master stream tasks.
S303, in the corresponding computer room of master stream tasks performs master stream task process, control master stream tasks
Corresponding computer room stores the execution state of master stream tasks to external data base or file system according to prefixed time interval.
S304 when detecting that master stream tasks occur abnormal in the process of implementation, is selected in follower stream tasks
It selects a stream task and flows task as current master.
S305 controls the corresponding computer room of current master streams task to perform current master streams task.
In the present embodiment, above-mentioned steps S301-S305 can respectively with the step S201-S205 in embodiment illustrated in fig. 2
Identical, which is not described herein again.
S306, in the corresponding computer room of current master stream tasks performs current master streams task process, control is current
The corresponding computer room of master stream tasks stores the execution state of current master stream tasks to outside according to prefixed time interval
Database or file system.
It, can in the corresponding computer room of current master stream tasks performs current master streams task process in the present embodiment
To control the corresponding computer room of current master stream task according to prefixed time interval by the execution state of current master stream tasks
It stores to external data base or file system, in this way, when current master stream tasks occur in the process of implementation such as step S103
In exception when, taking over any follower stream tasks of master stream tasks can know that current master stream task is current
Execution state, and stream task is performed since the execution state, without from the beginning performing entire stream task.
It is appreciated that step S306 is performed while step S305 is performed.
Stream task implementation method provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Fig. 4 be 4th kind provided in an embodiment of the present invention stream task implementation method flow diagram, this method include with
Lower step：
S401, select in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms a stream task as
Master flows task, and other stream tasks in multiple stream tasks in addition to master stream tasks are appointed as follower streams
Business.
S402, the corresponding computer room of control master stream tasks perform master stream tasks.
S403 when detecting that master stream tasks occur abnormal in the process of implementation, is selected in follower stream tasks
It selects a stream task and flows task as current master.
In the present embodiment, step S401-S403 can be identical with the step S101-S103 in Fig. 1 embodiments respectively,
Which is not described herein again.
S404, after detecting that master stream task recoveries are normal, will restore normal master stream task as
Follower flows task.
By step S403 it is found that when master stream tasks occur abnormal in the process of implementation, it is meant that master streams are appointed
Be engaged in corresponding computer room cannot perform master stream task, then the computer room exited stream task service.If it is appreciated that some
Computer room has exited stream task service, and the several of cluster for occurring network failure or computer room storage especially for the computer room set
Standby situation about breaking down, room management personnel can safeguard the computer room.
After detecting that the corresponding computer room of master stream tasks restores normal, it is meant that the stream of computer room computer room deployment is appointed
Business restores normal.It can will restore normal stream task and flow task as follower, in order to flow task in current master
When appearance is abnormal in the process of implementation, will restore normal stream task can be corresponding as current master stream tasks
One of follower stream tasks can therefrom select a stream task to switch over.
S405 controls the corresponding computer room of current master streams task to perform current master streams task.
In the present embodiment, step S405 can be identical with the step S104 in Fig. 1 embodiments, and which is not described herein again.
Stream task implementation method provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
It should be noted that in the embodiment of the method that any of the above-described stream task is realized, what is provided is once when master flows
There is the process that current master streams task is switched to during exception in the process of implementation in task, but in actual use, as long as working as
Before the master stream tasks that are carrying out occur abnormal, the above-mentioned stream task implementation method that can be provided by the application carries out
Master flows the switching of task, to ensure that entire stream tasks carrying is completed.
Fig. 5 is the structure diagram of the first stream task realization device provided in an embodiment of the present invention, which includes：The
One selecting module 501, the first control module 502, the second selecting module 503 and the second control module 504, wherein,
First choice module 501, for selecting one in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
A stream task as master flow task, and using in multiple stream tasks in addition to master stream tasks other stream tasks as
Follower flows task；
First control module 502, for the corresponding computer room of master stream tasks to be controlled to perform master stream tasks；
Second selecting module 503, for when detect master stream task occur abnormal in the process of implementation when,
A stream task is selected in follower stream tasks as current master and flows task；
Second control module 504 is appointed for the corresponding computer room of current master streams task to be controlled to perform current master streams
Business.
In a kind of realization method, first choice module 501, specifically for passing through Zookeeper in the advance portion of multiple computer rooms
A stream task is selected in the identical multiple stream tasks of the logic of administration as master stream tasks；
Second selecting module 503 selects a stream task specifically for passing through Zookeeper in follower stream tasks
Task is flowed as current master.
Stream task realization device provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Fig. 6 is the structure diagram of second provided in an embodiment of the present invention stream task realization device, which includes：The
One selecting module 601, the first control module 602, third control module 603, the second selecting module 604 and the second control module
605, wherein, first choice module 601, the first control module 602 and the second selecting module 604 can respectively with it is real shown in Fig. 5
First choice module 501, the first control module 502 applied in example are identical with the second selecting module 503, no longer heavy here
It is multiple.
In the present embodiment, third control module 603 is appointed for performing master streams in the corresponding computer room of master stream tasks
During business, the corresponding computer room of control master stream tasks deposits the execution state of master stream tasks according to prefixed time interval
It stores up to external data base or file system；
Second control module 605, for controlling the corresponding computer room of current master streams task from external data base or file
System reads the execution state of master stream tasks, and the state of current master stream tasks is restored to execution state, performs
Current master after state is restored flows task.
Stream task realization device provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Fig. 7 is the structure diagram of the third stream task realization device provided in an embodiment of the present invention, which includes：The
One selecting module 701, the first control module 702, third control module 703, the second selecting module 704, the second control module 705
With the 4th control module 706, wherein, first choice module 701, the first control module 702, third control module 703, second select
Selecting module 704, the second control module 705 can control with the first choice module 601, first in embodiment illustrated in fig. 6 respectively
Module 602, third control module 603, the second selecting module 604 are identical with the second control module 605, no longer heavy here
It is multiple.
In the present embodiment, the 4th control module 706 is current for being performed in the corresponding computer room of current master stream tasks
In master stream task process, control the corresponding computer room of current master stream task according to prefixed time interval by current master
The execution state of stream task is stored to external data base or file system.
Stream task realization device provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
Fig. 8 is the structure diagram of 4th kind provided in an embodiment of the present invention stream task realization device, which includes：The
One selecting module 801, the first control module 802, the second selecting module 803, recovery module 804, the second control module 805,
In, first choice module 801, the first control module 802, the second selecting module 803, the second control module 805 can respectively with
First choice module 501, the first control module 502, the second selecting module 503 and the second control module 504 in Fig. 5 embodiments
It is identical, it is not repeated herein.
In the present embodiment, recovery module 804, for selecting a stream in follower stream tasks in the second selecting module
After task is as current master streams task, after detecting that master stream task recoveries are normal, it will restore normal
Master flows task and flows task as follower.
Stream task realization device provided in an embodiment of the present invention passes through identical more of the logic disposed in advance in multiple computer rooms
Selected in a stream task one stream task as master stream task, and by multiple stream tasks in addition to master stream tasks
Other stream tasks flow task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；Work as inspection
Master stream tasks are measured when occurring abnormal in the process of implementation, one stream task select in follower stream tasks as ought
Preceding master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The present invention is implemented
Example when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
The embodiment of the present invention additionally provides a kind of electronic equipment, as shown in figure 9, including processor 901, communication interface 902,
Memory 903 and communication bus 904, wherein, processor 901, communication interface 902, memory 903 is complete by communication bus 904
Into mutual communication,
Memory 903, for storing computer program；
Processor 901 during for performing the program stored on memory 903, realizes following steps：
A stream task is selected to be flowed as master in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Task, and other stream tasks in multiple stream tasks in addition to master stream tasks are flowed into task as follower；
The corresponding computer room of master stream tasks is controlled to perform master stream tasks；
When detecting that master stream tasks occur abnormal in the process of implementation, one is selected in follower stream tasks
Stream task flows task as current master；
The corresponding computer room of current master streams task is controlled to perform current master streams task.
Electronic equipment provided in an embodiment of the present invention is appointed by the identical multiple streams of the logic disposed in advance in multiple computer rooms
Selected in business one stream task as master flow task, and by multiple stream tasks in addition to master stream tasks other flow
Task flows task as follower；The corresponding computer room of master stream tasks is controlled to perform master stream tasks；When detecting
When master stream tasks occur abnormal in the process of implementation, a stream task is selected in follower stream tasks as current
Master flows task；The corresponding computer room of current master streams task is controlled to perform current master streams task.The embodiment of the present invention
It when the cluster for running stream task breaks down, can ensure the normal operation of stream task.
The communication bus that above-mentioned electronic equipment is mentioned can be Peripheral Component Interconnect standard (Peripheral Component
Interconnect, abbreviation PCI) bus or expanding the industrial standard structure (Extended Industry Standard
Architecture, abbreviation EISA) bus etc..The communication bus can be divided into address bus, data/address bus, controlling bus etc..
For ease of representing, only represented in figure with a thick line, it is not intended that an only bus or a type of bus.
Communication interface is for the communication between above-mentioned electronic equipment and other equipment.
Memory can include random access memory (Random Access Memory, abbreviation RAM), can also include
Nonvolatile memory (non-volatile memory), for example, at least a magnetic disk storage.Optionally, memory may be used also
To be at least one storage device for being located remotely from aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit (Central Processing Unit,
Abbreviation CPU), network processing unit (Network Processor, abbreviation NP) etc.；It can also be digital signal processor
(Digital Signal Processing, abbreviation DSP), application-specific integrated circuit (Application Specific
Integrated Circuit, abbreviation ASIC), field programmable gate array (Field-Programmable Gate Array,
Abbreviation FPGA) either other programmable logic device, discrete gate or transistor logic, discrete hardware components.
In another embodiment provided by the invention, a kind of computer readable storage medium is additionally provided, which can
It reads to be stored with instruction in storage medium, when run on a computer so that computer performs any in above-described embodiment
Flow task implementation method.
The instruction of computer readable storage medium memory storage provided in an embodiment of the present invention, when it runs on computers
When, by the way that a stream task is selected to appoint as master streams in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Business, and other stream tasks in multiple stream tasks in addition to master stream tasks are flowed into task as follower；Control
The corresponding computer room of master stream tasks performs master stream tasks；When detect master stream task occur in the process of implementation it is different
Chang Shi selects a stream task as current master in follower stream tasks and flows task；Current master streams is controlled to appoint
Be engaged in corresponding computer room performs current master streams task.The embodiment of the present invention can break down in the cluster of operation stream task
When, the normal operation of guarantee stream task.
In another embodiment provided by the invention, a kind of computer program product for including instruction is additionally provided, when it
When running on computers so that computer performs stream task implementation method any in above-described embodiment.
It is provided in an embodiment of the present invention comprising instruction computer program product, when run on a computer, by
A stream task is selected to flow task as master, and will be more in the identical multiple stream tasks of logic that multiple computer rooms are disposed in advance
Other stream tasks in a stream task in addition to master stream tasks flow task as follower；Control master stream tasks pair
The computer room answered performs master stream tasks；When detecting that master stream tasks occur abnormal in the process of implementation,
A stream task is selected in follower stream tasks as current master and flows task；Control current master streams task corresponding
Computer room performs current master streams task.The embodiment of the present invention can ensure stream when the cluster for running stream task breaks down
The normal operation of task.
In the above-described embodiments, can come wholly or partly by software, hardware, firmware or its arbitrary combination real
It is existing.When implemented in software, it can entirely or partly realize in the form of a computer program product.The computer program
Product includes one or more computer instructions.When loading on computers and performing the computer program instructions, all or
It partly generates according to the flow or function described in the embodiment of the present invention.The computer can be all-purpose computer, special meter
Calculation machine, computer network or other programmable devices.The computer instruction can be stored in computer readable storage medium
In or from a computer readable storage medium to another computer readable storage medium transmit, for example, the computer
Instruction can pass through wired (such as coaxial cable, optical fiber, number from a web-site, computer, server or data center
User's line (DSL)) or wireless (such as infrared, wireless, microwave etc.) mode to another web-site, computer, server or
Data center is transmitted.The computer readable storage medium can be any usable medium that computer can access or
It is the data storage devices such as server, the data center integrated comprising one or more usable mediums.The usable medium can be with
It is magnetic medium, (for example, floppy disk, hard disk, tape), optical medium (for example, DVD) or semiconductor medium (such as solid state disk
Solid State Disk (SSD)) etc..
Each embodiment in this specification is described using relevant mode, identical similar portion between each embodiment
Point just to refer each other, and the highlights of each of the examples are difference from other examples.Especially for device/
For electronic equipment/storage medium/computer program product embodiments, since it is substantially similar to embodiment of the method, so retouching
That states is fairly simple, and the relevent part can refer to the partial explaination of embodiments of method.
It should be noted that herein, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any this practical relationship or sequence.Moreover, term " comprising ", "comprising" or its any other variant are intended to
Non-exclusive inclusion, so that process, method, article or equipment including a series of elements not only will including those
Element, but also including other elements that are not explicitly listed or further include as this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence " including one ... ", it is not excluded that
Also there are other identical elements in the process, method, article or apparatus that includes the element.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (11)
1. a kind of stream task implementation method, which is characterized in that including：
A stream task is selected to flow task as master in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms,
And other stream tasks in the multiple stream task in addition to the master streams task are flowed into task as follower；
The corresponding computer room of the master streams task is controlled to perform the master streams task；
When detecting that the master streams task occurs abnormal in the process of implementation, selected in the follower streams task
One stream task flows task as current master；
The corresponding computer room of the current master streams task is controlled to perform the current master streams task.
2. according to the method described in claim 1, it is characterized in that, the method further includes：
In the corresponding computer room of master stream tasks performs the master streams task process, the master streams is controlled to appoint
Be engaged in corresponding computer room according to prefixed time interval by the master stream task execution state store to external data base or text
Part system；
The control current master flows the corresponding computer room of task and performs the current master streams task, including：
The corresponding computer room of the current master streams task is controlled from described in the external data base or file system reading
Master flows the execution state of task, and the state of the current master streams task is restored to the execution state, performs
The current master after state is restored flows task.
3. according to the method described in claim 1, it is characterized in that, the method further includes：
In the current master flows the corresponding computer room execution of the task current master streams task process, control is described ought
Preceding master flows the corresponding computer room of task and stores the execution state of the current master streams task according to prefixed time interval
To external data base or file system.
4. according to the method described in claim 1, it is characterized in that, described select a stream in the follower streams task
After the step of task flows task as current master, the method further includes：
After detecting that master stream task recovery is normal, will restore the normal master stream tasks as
Follower flows task.
5. according to claim 1-4 any one of them methods, which is characterized in that
A stream task is selected to be flowed as master in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
Task, including：
Selected in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms by Zookeeper a stream task as
Master flows task；
It is described selected in the follower streams task one stream task as current master flow task, including：
A stream task is selected to appoint as current master streams in the follower streams task by the Zookeeper
Business.
6. a kind of stream task realization device, which is characterized in that including：
First choice module, for selecting a stream task in the identical multiple stream tasks of the logic disposed in advance in multiple computer rooms
As master flow task, and using in the multiple stream task in addition to the master streams task other stream tasks as
Follower flows task；
First control module, for the corresponding computer room of the master streams task to be controlled to perform the master streams task；
Second selecting module, for when detecting that master stream task occurs abnormal in the process of implementation, described
A stream task is selected in follower stream tasks as current master and flows task；
Second control module is appointed for the corresponding computer room of the current master streams task to be controlled to perform the current master streams
Business.
7. device according to claim 6, which is characterized in that described device further includes：
Third control module, used in performing the master streams task process in the corresponding computer room of master stream tasks,
The corresponding computer room of the master streams task is controlled to deposit the execution state of master stream tasks according to prefixed time interval
It stores up to external data base or file system；
Second control module, specifically for controlling the corresponding computer room of the current master streams task from the external data
Library or file system read the execution state of the master streams task, and the state of the current master streams task is restored
To the execution state, the current master streams task after state is restored is performed.
8. device according to claim 6, which is characterized in that described device further includes：
4th control module performs the current master streams task for flowing the corresponding computer room of task in the current master
In the process, the corresponding computer room of the current master streams task is controlled to appoint the current master streams according to prefixed time interval
The execution state of business is stored to external data base or file system.
9. device according to claim 6, which is characterized in that described device further includes：
Recovery module, for second selecting module selected in the follower streams task one stream task as ought
After preceding master streams task, after detecting that the master streams task recovery is normal, the normal master will be restored
Stream task flows task as follower.
10. according to the device described in claim 6-9, which is characterized in that
The first choice module, specifically for passing through Zookeeper in identical multiple of the logic that multiple computer rooms are disposed in advance
A stream task is selected in stream task as master and flows task；
Second selecting module, specifically for selecting one in the follower streams task by the Zookeeper
Stream task flows task as current master.
11. a kind of electronic equipment, which is characterized in that including processor, communication interface, memory and communication bus, wherein, processing
Device, communication interface, memory complete mutual communication by communication bus；
Memory, for storing computer program；
Processor during for performing the program stored on memory, realizes any method and steps of claim 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810010283.6A CN108228328B (en) | 2018-01-05 | 2018-01-05 | Stream task implementation method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810010283.6A CN108228328B (en) | 2018-01-05 | 2018-01-05 | Stream task implementation method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108228328A true CN108228328A (en) | 2018-06-29 |
| CN108228328B CN108228328B (en) | 2020-11-06 |
Family
ID=62643087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810010283.6A Active CN108228328B (en) | 2018-01-05 | 2018-01-05 | Stream task implementation method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108228328B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110321209A (en) * | 2019-06-28 | 2019-10-11 | 北京奇艺世纪科技有限公司 | A kind of task data processing method, device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070116029A1 (en) * | 2003-12-15 | 2007-05-24 | Di Lodovico Hakim | Distributed medium access control for broadband access systems |
| CN102932210A (en) * | 2012-11-23 | 2013-02-13 | 北京搜狐新媒体信息技术有限公司 | Method and system for monitoring node in PaaS cloud platform |
| CN104753994A (en) * | 2013-12-27 | 2015-07-01 | 杭州海康威视系统技术有限公司 | Method and device for data synchronization based on cluster server system |
| CN105117286A (en) * | 2015-09-22 | 2015-12-02 | 北京大学 | Task scheduling and pipelining executing method in MapReduce |
| CN105955820A (en) * | 2016-04-20 | 2016-09-21 | 北京云宏信达信息科技有限公司 | Method, device, and system for job stream control |
-
2018
- 2018-01-05 CN CN201810010283.6A patent/CN108228328B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070116029A1 (en) * | 2003-12-15 | 2007-05-24 | Di Lodovico Hakim | Distributed medium access control for broadband access systems |
| CN102932210A (en) * | 2012-11-23 | 2013-02-13 | 北京搜狐新媒体信息技术有限公司 | Method and system for monitoring node in PaaS cloud platform |
| CN104753994A (en) * | 2013-12-27 | 2015-07-01 | 杭州海康威视系统技术有限公司 | Method and device for data synchronization based on cluster server system |
| CN105117286A (en) * | 2015-09-22 | 2015-12-02 | 北京大学 | Task scheduling and pipelining executing method in MapReduce |
| CN105955820A (en) * | 2016-04-20 | 2016-09-21 | 北京云宏信达信息科技有限公司 | Method, device, and system for job stream control |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110321209A (en) * | 2019-06-28 | 2019-10-11 | 北京奇艺世纪科技有限公司 | A kind of task data processing method, device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108228328B (en) | 2020-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105005532B (en) | A kind of system and method for automatic test application programming interfaces stability | |
| CN105229973B (en) | The composition of network and the visualization system of action | |
| CN110138837A (en) | Request processing method, device, computer equipment and storage medium | |
| CN106713487A (en) | Data synchronization method and device | |
| CN102158364A (en) | Time slice scheduling-based device polling method and device | |
| CN109960634A (en) | A kind of method for monitoring application program, apparatus and system | |
| CN108416449A (en) | A kind of O＆M method and apparatus | |
| CN113114491B (en) | Method, device and equipment for constructing network topology | |
| CN109412875A (en) | Zookeeper cluster automatic maintenance method and device | |
| CN112040506A (en) | Reliability evaluation method and device of wireless sensor network and storage medium | |
| CN112333249A (en) | Business service system and method | |
| CN102724272B (en) | A kind of backup method of TV station's service distribution data | |
| CN104765672A (en) | Error code monitoring method, device and equipment | |
| US20090125846A1 (en) | Facility to Show Secondary Relationships in a Topological Graph Using an Associated View | |
| CN108199889A (en) | Creation method, device, server and the storage medium of service chaining | |
| CN108228328A (en) | A kind of stream task implementation method, device and electronic equipment | |
| JP5292929B2 (en) | Monitoring device | |
| CN114531373A (en) | Node state detection method, node state detection device, equipment and medium | |
| CN109445975A (en) | Abnormality eliminating method, logic device and member device | |
| CN109271270A (en) | The troubleshooting methodology, system and relevant apparatus of bottom hardware in storage system | |
| CN109697128A (en) | A kind of thread maintaining method, device and electronic equipment | |
| CN105871659A (en) | Monitoring method and device of newly added server | |
| CN105591902A (en) | Main-standby switching method and device | |
| CN110333930A (en) | Digital Platform system | |
| JP5730397B2 (en) | Program creation device and programmable logic controller |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |