CN108228328B - Stream task implementation method and device and electronic equipment - Google Patents
Stream task implementation method and device and electronic equipment Download PDFInfo
- Publication number
- CN108228328B CN108228328B CN201810010283.6A CN201810010283A CN108228328B CN 108228328 B CN108228328 B CN 108228328B CN 201810010283 A CN201810010283 A CN 201810010283A CN 108228328 B CN108228328 B CN 108228328B
- Authority
- CN
- China
- Prior art keywords
- task
- flow
- master
- flow task
- stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a method and a device for realizing a stream task and electronic equipment, wherein the method comprises the following steps: selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as folower flow tasks; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for implementing a streaming task, and an electronic device.
Background
The distributed real-time Streaming task framework, such as Spark Streaming, Flink, Storm, etc., can implement large-scale real-time Streaming data processing. When real-time streaming data processing is realized, when an error occurs in a certain machine or a plurality of machines and is down, the distributed real-time streaming task framework reschedules the task on the machine with the error to a normal machine to run so as to ensure the normal running of the streaming task.
However, the inventor finds that the prior art has at least the following problems in the process of implementing the invention: the existing distributed real-time streaming task framework cannot handle the situation that the whole cluster running the streaming task is unavailable, such as: 1. the cluster running the flow task is abnormal or needs to be stopped for maintenance; 2. the cluster running the streaming task is unavailable due to the error of the streaming task frame; 3. and (4) abnormal network of a machine room where the cluster of the running flow task is positioned, and the like. When these problems occur with a cluster running streaming tasks, the distributed real-time streaming task framework is not available.
Disclosure of Invention
The embodiment of the invention aims to provide a method, a device and electronic equipment for realizing a stream task, so as to ensure the normal operation of the stream task when a cluster for operating the stream task fails. The specific technical scheme is as follows:
in order to achieve the above object, in a first aspect, an embodiment of the present invention discloses a method for implementing a stream task, where the method includes:
selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as folower flow tasks;
controlling a machine room corresponding to the master flow task to execute the master flow task;
when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task;
and controlling the machine room corresponding to the current master flow task to execute the current master flow task.
Optionally, the method further includes:
in the process of executing the master flow task in the machine room corresponding to the master flow task, controlling the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval;
the step of controlling the machine room corresponding to the current master flow task to execute the current master flow task includes:
and controlling the machine room corresponding to the current master flow task to read the execution state of the master flow task from the external database or the file system, restoring the state of the current master flow task to the execution state, and executing the current master flow task after the state is restored.
Optionally, the method further includes:
and in the process of executing the current master flow task by the machine room corresponding to the current master flow task, controlling the machine room corresponding to the current master flow task to store the execution state of the current master flow task to an external database or a file system according to a preset time interval.
Optionally, after the step of selecting one of the follower flow tasks as the current master flow task, the method further includes:
and when the master flow task is detected to be recovered to be normal, taking the master flow task recovered to be normal as a follower flow task.
Optionally, the selecting one of the multiple flow tasks with the same logic and pre-deployed in multiple computer rooms as a master flow task includes:
selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of machine rooms through a Zookeeper as a master flow task;
selecting one flow task from the follower flow tasks as a current master flow task, wherein the selecting step comprises the following steps:
and selecting one flow task from the follower flow tasks as a current master flow task through the Zookeeper.
In a second aspect, an embodiment of the present invention discloses a device for implementing a streaming task, where the device includes:
the system comprises a first selection module, a second selection module and a third selection module, wherein the first selection module is used for selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task and taking other flow tasks except the master flow task from the plurality of flow tasks as follower flow tasks;
the first control module is used for controlling the machine room corresponding to the master flow task to execute the master flow task;
the second selection module is used for selecting one flow task from the follower flow tasks as the current master flow task when the fact that the master flow task is abnormal in the execution process is detected;
and the second control module is used for controlling the machine room corresponding to the current master flow task to execute the current master flow task.
Optionally, the apparatus further comprises:
the third control module is used for controlling the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval in the process of executing the master flow task by the machine room corresponding to the master flow task;
the second control module is specifically configured to control the machine room corresponding to the current master stream task to read the execution state of the master stream task from the external database or the file system, restore the state of the current master stream task to the execution state, and execute the current master stream task after the state is restored.
Optionally, the apparatus further comprises:
and the fourth control module is used for controlling the machine room corresponding to the current master flow task to store the execution state of the current master flow task to an external database or a file system according to a preset time interval in the process of executing the current master flow task by the machine room corresponding to the current master flow task.
Optionally, the apparatus further comprises:
and the recovery module is used for selecting one flow task from the follower flow tasks as a current master flow task by the second selection module, and taking the master flow task which is recovered to be normal as a follower flow task when the fact that the master flow task is recovered to be normal is detected.
Optionally, the first selection module is specifically configured to select, by Zookeeper, one stream task from a plurality of stream tasks with the same logic that are pre-deployed in a plurality of machine rooms as a master stream task;
the second selection module is specifically configured to select one stream task from the follower stream tasks as a current master stream task through the Zookeeper.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of the stream task implementation as described in the first aspect above when executing the program stored in the memory.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, which stores instructions that, when executed on a computer, cause the computer to execute the method steps of the streaming task implementation described in the first aspect.
In a fifth aspect, embodiments of the present invention also provide a computer program product containing instructions, which when run on a computer, cause the computer to perform the method steps of the streaming task implementation as described in the first aspect above.
According to the method, the device and the electronic equipment for realizing the stream tasks, provided by the embodiment of the invention, one stream task is selected from a plurality of stream tasks with the same logic, which are pre-deployed in a plurality of machine rooms, as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a schematic flowchart of a first method for implementing a streaming task according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a second method for implementing a streaming task according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a third method for implementing a streaming task according to an embodiment of the present invention;
fig. 4 is a flowchart illustrating a fourth flow task implementation method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a first flow task implementing device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a second flow task implementing device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a third flow task implementing device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a fourth flow task implementing device according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
Fig. 1 is a schematic flow chart of a first method for implementing a streaming task according to an embodiment of the present invention, where the method includes the following steps:
s101, selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as folower flow tasks.
In a distributed Streaming task framework, such as Spark Streaming, Flink, Storm, etc., a distributed Streaming task is divided into several tasks, and several machines in a cluster correspondingly execute the corresponding tasks respectively. Several machines in a cluster are typically stored in a machine room. When a problem occurs in a certain machine in the cluster, the distributed flow task framework can schedule the task to be executed or being executed by the machine with the problem to other machines in the normal working state in the cluster to continue executing.
For the case that the whole cluster fails during the execution of the distributed stream tasks, in this embodiment, multiple stream tasks with the same logic may be deployed in multiple machine rooms in advance, that is, the stream tasks with the same logic are deployed in each machine room. Logically identical streaming tasks refer to: the division rules and division results for dividing the stream task into a plurality of tasks are the same, and the data dependency relationships among different tasks are the same. In particular, the corresponding tasks may be performed by several machines stored in a cluster in each room.
For example, stream task A is divided into: task a, task b, and task c may deploy logically identical flow task a in 3 rooms, that is, room 1, room 2, and room 3. Specifically, the machines 11, 12, 13 in the cluster 1 stored in the machine room 1 sequentially execute tasks a, b, c, respectively; similarly, the machines 21, 22, 23 in the cluster 2 stored in the machine room 2 sequentially execute tasks a, b, c, respectively; the machines 31, 32, 33 in the cluster 3 stored in the machine room 3 execute tasks a, b, c in sequence, respectively.
It should be noted that, logical same flow tasks are deployed in multiple computer rooms, which does not mean that each computer room executes the flow tasks in the same time period, and actually, only one computer room executes the flow tasks in the same time period. Therefore, before the flow task is not started, one flow task needs to be selected from logically identical flow tasks pre-deployed in a plurality of computer rooms for execution, and the flow task selected for execution may be referred to as a master flow task.
In one implementation, selecting one stream task from a plurality of stream tasks with the same logic and pre-deployed in a plurality of computer rooms as a master stream task may include:
and selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms through the Zookeeper as a master flow task.
Zookeeper is a distributed application coordination service, and can provide a consistency service for distributed applications, and the provided service includes: configuration maintenance, domain name service, distributed synchronization, group service, etc. The core mechanisms of general Zookeeper include: a main flow and a synchronization flow are selected. When the service is just started, the leader is crashed and the follower is less than half, the system enters the main selection process, and the service is not provided to the outside at the moment; when the leader is elected, the system enters a synchronization process, the state synchronization among the services is completed, and then the external services are provided. In the main flow of selection, a commonly used algorithm is the fashleader election algorithm.
In this embodiment, before the streaming task is not started, an election mechanism based on Zookeeper may be used, specifically, a streaming task is selected as a master streaming task to be selectively executed from a plurality of streaming tasks with the same logic, which are pre-deployed in a plurality of machine rooms through a fashleader electric algorithm, which is equivalent to the election mechanism based on Zookeeper, and primary election is performed in the plurality of streaming tasks. The FashLeaderElection algorithm described above belongs to the prior art, and the embodiment will not be described in detail herein.
It will be appreciated that after selecting one of the plurality of stream tasks as the master stream task, then the other of the plurality of stream tasks other than the master stream task may be used as the follower stream task. The follower flow task may participate in voting in the Zookeeper's voting process, but after the master flow task is selected, the follower flow task does not perform a specific task.
And S102, controlling the machine room corresponding to the master flow task to execute the master flow task.
After the master flow task is selected, the machine room corresponding to the master flow task can be controlled to execute the master flow task according to the division result of the division task of the pre-flow task and the data dependency relationship among different tasks. Specifically, the corresponding tasks may be executed by a plurality of machines stored in the cluster of the machine room corresponding to the master stream task according to the division result and the execution order.
For example, in the example of step S101, if the flow task a deployed by the machine room 1 is selected as the master flow task, then the tasks a, b, c may be executed in sequence by the machines 11, 12, 13 in the cluster 1 stored by the machine room 1, respectively.
Certainly, when the machine room corresponding to the master flow task executes the master flow task, the machine room corresponding to the follower flow task does not execute the corresponding task, that is, the follower flow task is in a standby state. In step S101, specifically, when the master stream task is selected by the Zookeeper, the follower stream task may be used as a listener to monitor information from the Zookeeper about a change in the execution state of the stream task during the period when the machine room corresponding to the master stream task executes the master stream task. That is, when the execution state of the streaming task changes, the Zookeeper notifies the follower streaming task of the execution state change information of the streaming task. It can be understood that, specifically, the Zookeeper notifies the execution state change information of the flow task to the machine room corresponding to the follower flow task.
For example, in the example of step S101, if the flow task a deployed in the room 1 is selected as the master flow task, the flow tasks a deployed in the rooms 2 and 3 may be selected as the follower flow task.
S103, when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task.
In this embodiment, the occurrence of an exception in the execution process of the master stream task may include: 1. the cluster running the flow task is abnormal or needs to be stopped for maintenance; 2. the cluster running the streaming task is unavailable due to the error of the streaming task frame; 3. and (4) abnormal network of a machine room where the cluster of the running flow task is positioned, and the like.
When the master flow task is abnormal in the execution process, in order to ensure the normal execution of the flow task, one flow task can be selected from the follower flow tasks in the standby state as the current master flow task, so that the machine room corresponding to the current master flow task continues to execute the flow task.
In one implementation, selecting one of the follower flow tasks as the current master flow task may include:
and selecting one flow task from the follower flow tasks through the Zookeeper as the current master flow task.
In this implementation, the method is equivalent to an election mechanism based on Zookeeper, and performs secondary election in currently available followstream tasks in a standby state. For a specific implementation process, reference may be made to the implementation process of the first time main selection in step S101, which is not described herein again.
For example, in the example of step S101, the flow task a deployed by the room 2 is selected as the current master flow task.
And S104, controlling the machine room corresponding to the current master flow task to execute the current master flow task.
After the current master flow task is selected, the machine room corresponding to the current master flow task can be controlled to execute the current master flow task according to the division result of the division task of the pre-flow task and the data dependency relationship among different tasks. Specifically, the corresponding tasks may be executed by a plurality of machines stored in the cluster of the machine room corresponding to the current master stream task according to the division result and the execution order.
For example, in the example of step S101, if the flow task a deployed by the room 2 is selected as the current master flow task, the flow task a may be executed by the cluster 1 hosted by the room 2.
According to the method for realizing the stream task, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Fig. 2 is a schematic flow chart of a second method for implementing a streaming task according to an embodiment of the present invention, where the method includes the following steps:
s201, selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as follower flow tasks.
And S202, controlling a machine room corresponding to the master flow task to execute the master flow task.
In this embodiment, the steps S201 to S202 may be respectively identical to the steps S101 to S102 in the embodiment shown in fig. 1, and are not described herein again.
And S203, controlling the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval in the process of executing the master flow task by the machine room corresponding to the master flow task.
The execution state of the master stream task may include: which data the master stream task has currently processed, which data is not currently processed, etc. The method can also comprise the following steps: the master stream task is specifically which divided task is executed, and the like.
The external database may include: MySQL database, etc.
In the process that the machine room corresponding to the master stream task executes the master stream task, the machine room corresponding to the master stream task can be controlled to store the execution state of the master stream task to an external database or a file system according to the preset time interval, so that when the master stream task is abnormal in the execution process in the step S103, any follower stream task which replaces the master stream task can acquire the current execution state of the master stream task and start to execute the stream task from the execution state, and the whole stream task does not need to be executed from the beginning. The preset time interval can be set according to industrial needs.
It is understood that step S203 is performed at the same time step S202 is performed.
S204, when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task.
In this embodiment, the step S204 may be respectively identical to the step S103 in the embodiment shown in fig. 1, and is not described herein again.
And S205, controlling the machine room corresponding to the current master flow task to execute the current master flow task.
In this embodiment, controlling the machine room corresponding to the current master flow task to execute the current master flow task may include: and controlling a machine room corresponding to the current master stream task to read the execution state of the master stream task from an external database or a file system, restoring the state of the current master stream task to the execution state, and executing the current master stream task after the state is restored.
For example, the stream task a deployed in the machine room 1 is a master stream task, and the stream task a is divided into: task a, task b, and task c, in step S203, the machine room 1 stores the execution state of the master stream task to an external database or a file system according to a preset time interval. Suppose that the current sending and executing state of the master flow task is that the machine 11 in the cluster 1 stored in the machine room 1 executes the task a completely. And if the current master stream task is the stream task A deployed in the machine room 2, the machine room 2 reads the current execution state of the master stream task from an external database or a file system, and restores the state of the current master stream task to the execution state, namely, the state of the current master stream task is restored to the state that the execution task a is finished, and then the machine room 2 executes the current master stream task after the state is restored. Specifically, the machines 22 in the cluster 2 stored in the room 2 start to perform task b.
According to the method for realizing the stream task, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Fig. 3 is a flowchart of a third method for implementing a stream task according to an embodiment of the present invention, where the method includes the following steps:
s301, selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as folower flow tasks.
And S302, controlling a machine room corresponding to the master flow task to execute the master flow task.
And S303, controlling the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval in the process of executing the master flow task by the machine room corresponding to the master flow task.
S304, when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task.
S305, controlling the machine room corresponding to the current master flow task to execute the current master flow task.
In this embodiment, the steps S301 to S305 may be respectively identical to the steps S201 to S205 in the embodiment shown in fig. 2, and are not described herein again.
And S306, controlling the machine room corresponding to the current master flow task to store the execution state of the current master flow task to an external database or a file system according to a preset time interval in the process of executing the current master flow task by the machine room corresponding to the current master flow task.
In this embodiment, in the process of executing the current master stream task in the machine room corresponding to the current master stream task, the machine room corresponding to the current master stream task may be controlled to store the execution state of the current master stream task to the external database or the file system according to the preset time interval, so that when the current master stream task is abnormal in the execution process as in step S103, any follower stream task that takes over the master stream task may obtain the current execution state of the current master stream task and start executing the stream task from the execution state, without executing the whole stream task from the beginning.
It is understood that step S306 is performed at the same time as step S305 is performed.
According to the method for realizing the stream task, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Fig. 4 is a flowchart illustrating a fourth flow task implementation method according to an embodiment of the present invention, where the method includes the following steps:
s401, selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as folower flow tasks.
And S402, controlling the machine room corresponding to the master flow task to execute the master flow task.
S403, when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task.
In this embodiment, steps S401 to S403 may be respectively identical to steps S101 to S103 in the embodiment of fig. 1, and are not described herein again.
S404, when the master flow task is detected to be recovered to be normal, the recovered master flow task is used as a follower flow task.
Step S403 shows that when the master stream task is abnormal in the execution process, it means that the machine room corresponding to the master stream task cannot execute the master stream task, and the machine room exits the stream task service. It can be understood that if a certain machine room exits from the stream task service, especially in case of a network failure of the machine room or a failure of several devices of a cluster stored in the machine room, a machine room maintenance person may perform maintenance on the machine room.
When the fact that the machine room corresponding to the master flow task is recovered to be normal is detected, the fact that the flow task deployed in the machine room is recovered to be normal means. The flow task which is recovered to be normal can be used as a follower flow task, so that when the current master flow task is abnormal in the execution process, the flow task which is recovered to be normal can be used as one of the follower flow tasks corresponding to the current master flow task, and one flow task can be selected from the flow tasks to be switched.
And S405, controlling the machine room corresponding to the current master flow task to execute the current master flow task.
In this embodiment, step S405 may be completely the same as step S104 in the embodiment of fig. 1, and is not described herein again.
According to the method for realizing the stream task, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
It should be noted that, in the embodiment of the method for implementing any flow task, a process of switching to a current master flow task once when the master flow task is abnormal in the execution process is provided, but in actual use, as long as the currently executed master flow task is abnormal, the master flow task can be switched by using the flow task implementation method provided by the present application, so as to ensure that the execution of the whole flow task is completed.
Fig. 5 is a schematic structural diagram of a first flow task implementing device provided in an embodiment of the present invention, where the device includes: a first selection module 501, a first control module 502, a second selection module 503, and a second control module 504, wherein,
a first selecting module 501, configured to select one stream task from multiple stream tasks with the same logic, which are pre-deployed in multiple computer rooms, as a master stream task, and use other stream tasks except the master stream task in the multiple stream tasks as a follower stream task;
the first control module 502 is used for controlling a machine room corresponding to the master flow task to execute the master flow task;
a second selecting module 503, configured to select one of the workflow tasks as a current master workflow task when it is detected that the master workflow task is abnormal in the execution process;
and the second control module 504 is configured to control the machine room corresponding to the current master flow task to execute the current master flow task.
In an implementation manner, the first selecting module 501 is specifically configured to select one stream task from a plurality of stream tasks with the same logic, which are pre-deployed in a plurality of machine rooms through a Zookeeper, as a master stream task;
the second selecting module 503 is specifically configured to select one stream task from the follower stream tasks as the current master stream task through the Zookeeper.
According to the device for realizing the stream task, provided by the embodiment of the invention, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Fig. 6 is a schematic structural diagram of a second flow task implementing device according to an embodiment of the present invention, where the device includes: the first selection module 601, the first control module 602, the third control module 603, the second selection module 604 and the second control module 605, wherein the first selection module 601, the first control module 602 and the second selection module 604 may be identical to the first selection module 501, the first control module 502 and the second selection module 503 in the embodiment shown in fig. 5, respectively, and are not repeated here.
In this embodiment, the third control module 603 is configured to control the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval in the process that the machine room corresponding to the master flow task executes the master flow task;
the second control module 605 is configured to control the machine room corresponding to the current master stream task to read the execution state of the master stream task from the external database or the file system, restore the state of the current master stream task to the execution state, and execute the current master stream task after the state is restored.
According to the device for realizing the stream task, provided by the embodiment of the invention, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Fig. 7 is a schematic structural diagram of a third streaming task implementing device according to an embodiment of the present invention, where the device includes: the first selection module 701, the first control module 702, the third control module 703, the second selection module 704, the second control module 705 and the fourth control module 706, wherein the first selection module 701, the first control module 702, the third control module 703, the second selection module 704 and the second control module 705 may be respectively identical to the first selection module 601, the first control module 602, the third control module 603, the second selection module 604 and the second control module 605 in the embodiment shown in fig. 6, and are not repeated here.
In this embodiment, the fourth control module 706 is configured to control the machine room corresponding to the current master flow task to store the execution state of the current master flow task to an external database or a file system according to a preset time interval in the process that the machine room corresponding to the current master flow task executes the current master flow task.
According to the device for realizing the stream task, provided by the embodiment of the invention, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
Fig. 8 is a schematic structural diagram of a fourth flow task implementing device according to an embodiment of the present invention, where the device includes: a first selection module 801, a first control module 802, a second selection module 803, a recovery module 804, and a second control module 805, wherein the first selection module 801, the first control module 802, the second selection module 803, and the second control module 805 may be identical to the first selection module 501, the first control module 502, the second selection module 503, and the second control module 504, respectively, in the embodiment of fig. 5, and are not repeated here.
In this embodiment, the recovering module 804 is configured to, after the second selecting module selects one of the workflow tasks as the current master workflow task, and when it is detected that the master workflow task is recovered to be normal, use the recovered normal master workflow task as the workflow task.
According to the device for realizing the stream task, provided by the embodiment of the invention, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
An embodiment of the present invention further provides an electronic device, as shown in fig. 9, which includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, where the processor 901, the communication interface 902, and the memory 903 complete mutual communication through the communication bus 904,
a memory 903 for storing computer programs;
the processor 901 is configured to implement the following steps when executing the program stored in the memory 903:
selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as a follower flow task;
controlling a machine room corresponding to the master flow task to execute the master flow task;
when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task;
and controlling the machine room corresponding to the current master flow task to execute the current master flow task.
According to the electronic equipment provided by the embodiment of the invention, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of machine rooms to serve as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are taken as a follower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, which stores instructions that, when executed on a computer, cause the computer to perform the method for implementing the streaming task in any one of the above embodiments.
When the instruction stored in the computer-readable storage medium provided by the embodiment of the invention runs on a computer, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of computer rooms as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are used as a folower stream task; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
In yet another embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the streaming task implementation method of any of the above embodiments.
When the computer program product containing the instruction provided by the embodiment of the invention runs on a computer, one stream task is selected from a plurality of stream tasks with the same logic and pre-deployed in a plurality of computer rooms as a master stream task, and other stream tasks except the master stream task in the plurality of stream tasks are used as folower stream tasks; controlling a machine room corresponding to the master flow task to execute the master flow task; when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task; and controlling the machine room corresponding to the current master flow task to execute the current master flow task. The embodiment of the invention can ensure the normal operation of the stream task when the cluster for operating the stream task fails.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus/electronic device/storage medium/computer program product embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "comprising", without further limitation, means that the element so defined is not excluded from the group consisting of additional identical elements in the process, method, article, or apparatus that comprises the element.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (11)
1. A method for implementing a streaming task, comprising:
selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task, and taking other flow tasks except the master flow task from the plurality of flow tasks as folower flow tasks;
controlling a machine room corresponding to the master flow task to execute the master flow task;
when the fact that the master flow task is abnormal in the execution process is detected, one flow task is selected from the follower flow tasks to serve as the current master flow task;
and controlling the machine room corresponding to the current master flow task to execute the current master flow task.
2. The method of claim 1, further comprising:
in the process of executing the master flow task in the machine room corresponding to the master flow task, controlling the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval;
the step of controlling the machine room corresponding to the current master flow task to execute the current master flow task includes:
and controlling the machine room corresponding to the current master flow task to read the execution state of the master flow task from the external database or the file system, restoring the state of the current master flow task to the execution state, and executing the current master flow task after the state is restored.
3. The method of claim 1, further comprising:
and in the process of executing the current master flow task by the machine room corresponding to the current master flow task, controlling the machine room corresponding to the current master flow task to store the execution state of the current master flow task to an external database or a file system according to a preset time interval.
4. The method of claim 1, wherein after the step of selecting one of the follower stream tasks as the current master stream task, the method further comprises:
and when the master flow task is detected to be recovered to be normal, taking the master flow task recovered to be normal as a follower flow task.
5. The method according to any one of claims 1 to 4,
the selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task comprises the following steps:
selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of machine rooms through a Zookeeper as a master flow task;
selecting one flow task from the follower flow tasks as a current master flow task, wherein the selecting step comprises the following steps:
and selecting one flow task from the follower flow tasks as a current master flow task through the Zookeeper.
6. A streaming task implementing apparatus, comprising:
the system comprises a first selection module, a second selection module and a third selection module, wherein the first selection module is used for selecting one flow task from a plurality of flow tasks with the same logic and pre-deployed in a plurality of computer rooms as a master flow task and taking other flow tasks except the master flow task from the plurality of flow tasks as follower flow tasks;
the first control module is used for controlling the machine room corresponding to the master flow task to execute the master flow task;
the second selection module is used for selecting one flow task from the follower flow tasks as the current master flow task when the fact that the master flow task is abnormal in the execution process is detected;
and the second control module is used for controlling the machine room corresponding to the current master flow task to execute the current master flow task.
7. The apparatus of claim 6, further comprising:
the third control module is used for controlling the machine room corresponding to the master flow task to store the execution state of the master flow task to an external database or a file system according to a preset time interval in the process of executing the master flow task by the machine room corresponding to the master flow task;
the second control module is specifically configured to control the machine room corresponding to the current master stream task to read the execution state of the master stream task from the external database or the file system, restore the state of the current master stream task to the execution state, and execute the current master stream task after the state is restored.
8. The apparatus of claim 6, further comprising:
and the fourth control module is used for controlling the machine room corresponding to the current master flow task to store the execution state of the current master flow task to an external database or a file system according to a preset time interval in the process of executing the current master flow task by the machine room corresponding to the current master flow task.
9. The apparatus of claim 6, further comprising:
and the recovery module is used for selecting one flow task from the follower flow tasks as a current master flow task by the second selection module, and taking the master flow task which is recovered to be normal as a follower flow task when the fact that the master flow task is recovered to be normal is detected.
10. The apparatus according to any one of claims 6 to 9,
the first selection module is specifically configured to select one stream task from a plurality of stream tasks with the same logic, which are pre-deployed in a plurality of machine rooms, as a master stream task through a Zookeeper;
the second selection module is specifically configured to select one stream task from the follower stream tasks as a current master stream task through the Zookeeper.
11. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810010283.6A CN108228328B (en) | 2018-01-05 | 2018-01-05 | Stream task implementation method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810010283.6A CN108228328B (en) | 2018-01-05 | 2018-01-05 | Stream task implementation method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108228328A CN108228328A (en) | 2018-06-29 |
| CN108228328B true CN108228328B (en) | 2020-11-06 |
Family
ID=62643087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810010283.6A Active CN108228328B (en) | 2018-01-05 | 2018-01-05 | Stream task implementation method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108228328B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110321209A (en) * | 2019-06-28 | 2019-10-11 | 北京奇艺世纪科技有限公司 | A kind of task data processing method, device and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932210A (en) * | 2012-11-23 | 2013-02-13 | 北京搜狐新媒体信息技术有限公司 | Method and system for monitoring node in PaaS cloud platform |
| CN104753994A (en) * | 2013-12-27 | 2015-07-01 | 杭州海康威视系统技术有限公司 | Method and device for data synchronization based on cluster server system |
| CN105117286A (en) * | 2015-09-22 | 2015-12-02 | 北京大学 | Task scheduling and pipelining executing method in MapReduce |
| CN105955820A (en) * | 2016-04-20 | 2016-09-21 | 北京云宏信达信息科技有限公司 | Method, device, and system for job stream control |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2003292242A1 (en) * | 2003-12-15 | 2005-07-05 | Telefonaktiebolaget L M Ericsson (Publ) | Distributed medium acces control for broadband access systems |
-
2018
- 2018-01-05 CN CN201810010283.6A patent/CN108228328B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932210A (en) * | 2012-11-23 | 2013-02-13 | 北京搜狐新媒体信息技术有限公司 | Method and system for monitoring node in PaaS cloud platform |
| CN104753994A (en) * | 2013-12-27 | 2015-07-01 | 杭州海康威视系统技术有限公司 | Method and device for data synchronization based on cluster server system |
| CN105117286A (en) * | 2015-09-22 | 2015-12-02 | 北京大学 | Task scheduling and pipelining executing method in MapReduce |
| CN105955820A (en) * | 2016-04-20 | 2016-09-21 | 北京云宏信达信息科技有限公司 | Method, device, and system for job stream control |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108228328A (en) | 2018-06-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109445927B (en) | Task management method and device for storage cluster | |
| CN106572137B (en) | Distributed service resource management method and device | |
| CN111092865A (en) | Security event analysis method and system | |
| CN112463437A (en) | Service recovery method, system and related components of storage cluster system offline node | |
| EP3591530B1 (en) | Intelligent backup and recovery of cloud computing environment | |
| CN108228328B (en) | Stream task implementation method and device and electronic equipment | |
| CN113126925B (en) | Member list determining method, device and equipment and readable storage medium | |
| CN109271270A (en) | The troubleshooting methodology, system and relevant apparatus of bottom hardware in storage system | |
| CN113672471A (en) | Software monitoring method, device, equipment and storage medium | |
| CN109525408B (en) | Equipment exception handling method and device and cloud storage system | |
| CN108154343B (en) | Emergency processing method and system for enterprise-level information system | |
| CN115687019A (en) | Database cluster fault processing method, intelligent monitoring platform, equipment and medium | |
| CN113778763B (en) | Intelligent switching method and system for three-way interface service faults | |
| CN110968456A (en) | Method and device for processing fault disk in distributed storage system | |
| CN115102862A (en) | Automatic synchronization method and device for SDN equipment | |
| CN110673793B (en) | Storage device node event management method and system, electronic device and storage medium | |
| CN110519393B (en) | Self-service equipment supervision method, device, equipment, server and medium | |
| CN111090491B (en) | Virtual machine task state recovery method and device and electronic equipment | |
| CN113010295A (en) | Stream computing method, device, equipment and storage medium | |
| CN116938881B (en) | Method, system, equipment and readable storage medium for realizing dynamic IP pool | |
| CN115934428B (en) | Main disaster recovery and backup switching method and device of MYSQL database and electronic equipment | |
| CN113568719B (en) | Service fault processing method and device, electronic equipment and storage medium | |
| CN114363333B (en) | Method, apparatus, device and medium for assigning machine ID in distributed system | |
| CN113760458A (en) | Node management method and device, electronic equipment and storage medium | |
| CN110677268B (en) | NFV automatic arrangement scheduling method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |