CA3168286A1 - Data flow processing method and system - Google Patents
Data flow processing method and systemInfo
- Publication number
- CA3168286A1 CA3168286A1 CA3168286A CA3168286A CA3168286A1 CA 3168286 A1 CA3168286 A1 CA 3168286A1 CA 3168286 A CA3168286 A CA 3168286A CA 3168286 A CA3168286 A CA 3168286A CA 3168286 A1 CA3168286 A1 CA 3168286A1
- Authority
- CA
- Canada
- Prior art keywords
- task
- node
- worker
- nodes
- plural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000005111 flow chemistry technique Methods 0.000 title claims description 19
- 230000008569 process Effects 0.000 claims description 19
- 238000012545 processing Methods 0.000 abstract description 7
- 230000000977 initiatory effect Effects 0.000 abstract description 2
- 238000003672 processing method Methods 0.000 abstract 1
- 239000002699 waste material Substances 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000011038 discontinuous diafiltration by volume reduction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A data stream processing method and system, belonging to the field of big data processing. The method comprises: a Zookeeper cluster determining one of several Master nodes to be a master node (101); the master node providing an external interface for receiving an online request of a service, and assigning a task to the service (102); the master node generating, according to current state information reported by each of a plurality of Worker nodes, configuration information of the task and writing same into the Zookeeper cluster, wherein the configuration information comprises information for indicating a Worker node scheduled to execute the task (103); and if the Worker node monitors that a task scheduled thereto is present in the Zookeeper cluster, initiating a Flume service for execution (104). The high availability of Master nodes and Worker nodes can be realized, the availability of a Flume service is improved, and the problems of unbalanced usage and the waste of resources are avoided; in addition, online and offline operations of a service can be greatly simplified, and mutual influence between services is reduced.
Description
DATA FLOW PROCESSING METHOD AND SYSTEM
BACKGROUND OF THE INVENTION
Technical Field [0001] The present invention relates to the field of big data processing technology, and more particularly to a data flow processing method and a data flow processing system.
Description of Related Art
BACKGROUND OF THE INVENTION
Technical Field [0001] The present invention relates to the field of big data processing technology, and more particularly to a data flow processing method and a data flow processing system.
Description of Related Art
[0002] It is usual in prior-art technology to start Flume service at various nodes of a cluster to unload data of the data source into the sink end.
[0003] However, as the inventor has found in the process of implementing the present invention, since the system cannot perceive conventional failures (such as deadlock, consumption abnormality, etc.) occurring to the node equipment, human processing is required, and timeliness in failure handling is adversely affected. Moreover, since various nodes of the same and single cluster are identically configured, but business data volumes of the various nodes differ from one another, this easily leads to larger proportion of idleness of Flume collection threads. Furthermore, since new businesses are relatively frequently handled online and it is required to manually modify business configuration files, the whole cluster should be restarted during such modification, whereby normal executions of other businesses in the same cluster are disturbed.
SUMMARY OF THE INVENTION
SUMMARY OF THE INVENTION
[0004] Aiming to solve at least one of the technical problems pending in the state of the art or in the relevant technologies, the present invention provides a data flow processing method Date Regue/Date Received 2022-07-18 and a data flow processing system.
[0005] Specific technical solutions provided by the embodiments of the present invention are recited below.
[0006] According to one aspect, there is provided a data flow processing method that comprises:
[0007] determining by a Zookeeper cluster one of plural Master nodes to serve as a leader node;
[0008] by the leader node, providing an external interface to receive a request to go online of a business and assigning a task to the business;
[0009] generating configuration information of the task and writing the same in the Zookeeper cluster according to current state information respectively reported by plural Worker nodes, wherein the configuration information includes information instructing a Worker node scheduled to execute the task; and
[0010] if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
[0011] Further, the step of determining by a Zookeeper cluster one of plural Master nodes to serve as a leader node includes:
[0012] receiving, by the Zookeeper cluster, a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
[0013] the Master nodes being started;
[0014] failure occurring to the current Master node serving as the leader node.
[0015] Moreover, the step of generating configuration information of the task according to current state information respectively reported by plural Worker nodes includes:
[0016] determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Date Regue/Date Received 2022-07-18 Worker nodes; and
[0017] generating configuration information instructing to schedule the task to the target Worker node.
[0018] Furthermore, the method further comprises:
[0019] adjusting by the leader node the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein
[0020] the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and
[0021] transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
[0022] Moreover, the method further comprises:
[0023] receiving by the leader node a request to go offline of the business through the external interface; and
[0024] writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
[0025] According to the other aspect, there is provided a data flow processing system that comprises a Zookeeper cluster, plural Master nodes, and plural Worker nodes, wherein
[0026] the Zookeeper cluster is employed for determining one of plural Master nodes to serve as the leader node;
[0027] the leader node is employed for providing an external interface to receive a request to go online of a business, and assigning a task to the business;
[0028] the leader node is further employed for generating configuration information of the task and writing the same in the Zookeeper cluster according to current state information Date Regue/Date Received 2022-07-18 respectively reported by plural Worker nodes, wherein the configuration information includes information instructing a Worker node scheduled to execute the task;
and
and
[0029] the Worker node is employed for if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
[0030] Further, the Zookeeper cluster is specifically employed for:
[0031] receiving a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
[0032] the Master nodes being started;
[0033] failure occurring to the current Master node serving as the leader node.
[0034] Moreover, the leader node is specifically employed for:
[0035] determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes; and
[0036] generating configuration information instructing to schedule the task to the target Worker node.
[0037] Furthermore, the leader node is specifically further employed for:
[0038] adjusting the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein
[0039] the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and
[0040] transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
Date Regue/Date Received 2022-07-18
Date Regue/Date Received 2022-07-18
[0041] Moreover, the leader node is specifically further employed for:
[0042] receiving a request to go offline of the business through the external interface; and
[0043] writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
[0044] The technical solutions provided by the embodiments of the present invention bring about the following advantageous effects.
[0045] 1. Through determining by a Zookeeper cluster one of plural Master nodes to serve as the leader node, the Master nodes achieve a high availability mechanism by means of the Zookeeper cluster, which guarantees that, when one of the Master nodes is problematic, another Master node takes over the external service in a short time, whereby availability of the Flume service is enhanced, and the prior-art problem, namely untimely processing of conventional failures occurring at the node equipment affects processing timeliness, is also solved at the same time.
[0046] 2. Through providing an external interface by the leader node, it is possible to facilitate the user to directly invoke the external interface of the leader node to put the task online/offline, which can shorten the operating time for putting the business online/offline within one minute, so as to greatly simplify the operation for putting the business online/offline; moreover, when configuration is updated, it is not required to manually modify business configuration file, and it is also not required to restart the cluster, as it is merely required to restart the business, so that mutual interference amongst businesses is reduced.
[0047] 3. By generating configuration information of the task and writing the same in the Zookeeper cluster by the leader node according to host running state information respectively reported by plural Worker nodes, and if the Worker node monitors a task Date Regue/Date Received 2022-07-18 scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task, unified administration of configurations by the Zookeeper cluster is realized, the problem of larger proportion of idleness of Flume collection threads is avoided, the problems concerning uneven utilization and wastage of resources are hence solved, and conveniency in operation and maintenance is also enhanced.
BRIEF DESCRIPTION OF THE DRAWINGS
BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to describe the technical solutions in the embodiments of the present invention more clearly, accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Apparently, the accompanying drawings introduced below are merely directed to partial embodiments of the present invention, and it is possible for persons ordinarily skilled in the art to acquire other drawings based on these drawings without spending any creative effort in the process.
[0049] Fig. 1 is a flowchart illustrating a data flow processing method provided by Embodiment 1 of the present invention; and
[0050] Fig. 2 is a block diagram illustrating a data flow processing system provided by Embodiment 2 of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
DETAILED DESCRIPTION OF THE INVENTION
[0051] In order to make more lucid and clear the objectives, technical solutions, and advantages of the present invention, the technical solutions in the embodiments of the present invention will be clearly and comprehensively described below with reference to accompanying drawings in the embodiments of the present invention. Apparently, the embodiments as described are merely partial, rather than the entire, embodiments of the present invention. All other embodiments obtainable by persons ordinarily skilled in the art based on the embodiments in the present invention shall all be covered by the Date Regue/Date Received 2022-07-18 protection scope of the present invention.
[0052] As should be understood, the terms "first" and "second", etc., as used in the description of the present application are merely meant for descriptive purposes, and shall not be understood to indicate or imply relative importance. Moreover, unless noted otherwise in the description of the present application, the word "plural" means two or more.
[0053] Before the embodiments of the present invention are introduced, several technical terms are briefly introduced below at first.
[0054] Zookeeper: a subproject ofHadoop, it is a reliable coordination system directed to a large-scale distributed system, and functions provided thereby include:
configuration maintenance, name service, distributed synchronization, and group service, etc.
configuration maintenance, name service, distributed synchronization, and group service, etc.
[0055] Flume: it is a highly available, highly reliable, and distributed technique for collecting, aggregating, and transmitting massive volume of logs.
[0056] An embodiment of the present invention provides a data flow processing method, the method can be executed by a data flow processing system, and the system is embodied as a distributed master-slave framework including a Zookeeper cluster, plural Master nodes, and plural Worker nodes, wherein the various Master nodes and various Worker nodes should be firstly registered with server nodes in the Zookeeper cluster, so that the various Master nodes and various Worker nodes can be unifounly administered via the Zookeeper cluster; the various Master nodes and various Worker nodes can be respectively configured to correspond to one hardware server, the specific numbers of the Master nodes and the Worker nodes can be decided by the user according to his own business application scenario, and no specific definition is made thereto in this context.
In the embodiments of the present invention, the various Worker nodes respectively start the Flume service, and take the Flume service as a data unloading tool responsible for Date Regue/Date Received 2022-07-18 unloading data of the data source into the sink end.
In the embodiments of the present invention, the various Worker nodes respectively start the Flume service, and take the Flume service as a data unloading tool responsible for Date Regue/Date Received 2022-07-18 unloading data of the data source into the sink end.
[0057] Fig. 1 is a flowchart illustrating a data flow processing method provided by Embodiment 1 of the present invention, as shown in Fig. 1, the method can comprise the following steps.
[0058] 101 ¨ determining by a Zookeeper cluster one of plural Master nodes to serve as the leader node.
[0059] There can be two Master nodes, when one of the Master nodes is determined to serve as the leader node, the other one is determined as a standby node. When a Master node is determined to serve as the leader node, the Master node can provide outgoing services, and provide an external interface for going-online, going-offline, checking and modifying of businesses, and is responsible for the scheduling of tasks.
[0060] Specifically, the ZooKeeper cluster receives a leader node election request initiated by a Master nodes on the basis of a preset triggering event, and takes the Master node as the leader node if the election is successful.
[0061] In an exemplary process of implementation, the preset triggering event can be the Master nodes being initiated.
[0062] Specifically speaking, after a Master node has executed an initiating command, the Master node initiates an election request for participating in the election of the leader node to the ZooKeeper cluster, if the ZooKeeper cluster determines that there is already an Active Master node serving as the leader node, the election of the Master node fails, and the ZooKeeper cluster records the state of this Master node as a Standby state; if the ZooKeeper cluster determines that there is no Active Master node serving as the leader node, the election of the Master node succeeds, and the ZooKeeper cluster records the Date Regue/Date Received 2022-07-18 state of this Master node as an Active state. The Master node in the Active state provides services to the outside.
[0063] In another exemplary process of implementation, the preset triggering event can be failure occurring to the current Master node serving as the leader node.
[0064] Specifically speaking, if failure occurs to the current Master node serving as the leader node, the ZooKeeper cluster obtains failure information of the current Master node, receives election requests initiated by other Master nodes except for the current Master node, and determines within a preset time one Master node from the other Master nodes to serve as the leader node, wherein the election process can be to elect a Master node with the best host performance from the other Master nodes to serve as the leader node, to which no definition is made in the embodiments of the present invention.
[0065] As should be noted, after one Master node has been elected to serve as the leader node, this Master node will start plural functional modules, and these plural functional modules can specifically include RestServer, RPC Server, and Scheduler, of which RestServer is responsible for adding, deleting, modifying, and checking businesses, RPCServer is responsible for receiving current state information reported by Worker nodes, and Scheduler is responsible for dynamically scheduling and assigning tasks.
[0066] Besides, the Zookeeper cluster also stores therein working directories that include, but are not limited to:
[0067] Leader, a temporary node for Master node election;
[0068] Master, a Service URL for disposing Active Master;
[0069] Jobs, directory nodes storing Job data;
[0070] Workers, temporary nodes for Worker node discovery;
[0071] Assign/<worker-id>/, a list to assign tasks, namely one directory node for each Worker node.
Date Regue/Date Received 2022-07-18
Date Regue/Date Received 2022-07-18
[0072] In the embodiments of the present invention, through determining by a Zookeeper cluster one of plural Master nodes to serve as the leader node, the Master nodes achieve a high availability mechanism by means of the Zookeeper cluster, which guarantees that, when one of the Master nodes is problematic, another Master node takes over the external service in a short time, and the prior-art problem, namely untimely processing of conventional failures occurring at the node equipment affects processing timeliness, is also solved at the same time.
[0073] 102 - providing by the leader node an external interface to receive a request to go online of a business, and assigning a task to the business.
[0074] Specifically, the leader node determined from plural Master nodes provides an external interface, the external interface can be used to receive a request to go online of a business (namely a Job), and the user can put the business online by invoking the external interface through a business application interface, wherein the request to go online of the business carries therewith configuration information of the business imported by means of a JSON
format; the leader node initializes business parameters according to the configuration information of the business, and assigns a task to the business, wherein the task is an execution unit of the Job on a Worker node, responsible for reading data from the source end and unloading the data to the sink end.
format; the leader node initializes business parameters according to the configuration information of the business, and assigns a task to the business, wherein the task is an execution unit of the Job on a Worker node, responsible for reading data from the source end and unloading the data to the sink end.
[0075] In the embodiments of the present invention, through providing an external interface by the leader node, it is possible to facilitate the user to directly invoke the external interface of the leader node to put the task online/offline, which can shorten the operating time for putting the business online/offline within one minute, so as to simplify the operation for putting the business online/offline; moreover, when configuration is updated, it is not required to manually modify business configuration file, and it is also not required to restart the cluster, as it is merely required to restart the business, so that mutual Date Regue/Date Received 2022-07-18 interference amongst businesses is reduced.
[0076] 103 - generating by the leader node configuration information of the task and writing the same in the ZooKeeper cluster according to host running state information respectively reported by plural Worker nodes, wherein the configuration information includes information instructing a Worker node scheduled to execute the task.
[0077] The host running state information includes one or more of CPU
utilization rate, memory utilization rate, magnetic disk reading/writing, and network uplink and downlink.
utilization rate, memory utilization rate, magnetic disk reading/writing, and network uplink and downlink.
[0078] Specifically, the process can include:
[0079] determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes; and generating configuration information instructing to schedule the task to the target Worker node.
[0080] In this embodiment, after the various worker nodes have been started, a Report thread will be started to be responsible for monitoring the running state of its own host, generating host running state information and reporting the same to the Master node serving as the leader node.
[0081] 104 - if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service and execute the task.
[0082] Specifically, the various Worker nodes respectively monitor the state in the ZooKeeper cluster, once a newly added task in the ZooKeeper cluster is monitored and the task is scheduled to the given Worker node itself, the Worker node obtains configuration information of the task from the ZooKeeper cluster, and starts the Flume service to execute the task, wherein Flume services are respectively deployed for each and every Date Regue/Date Received 2022-07-18 Worker node.
[0083] In the embodiments of the present invention, by generating configuration information of the task and writing the same in the Zookeeper cluster by the leader node, and if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service and execute the task, unified administration of configurations by the ZooKeeper cluster is realized, the problem of larger proportion of idleness of Flume collection threads is avoided, the problems concerning uneven utilization and wastage of resources are hence solved, and conveniency in operation and maintenance is also enhanced.
[0084] As a further preferred embodiment, the method provided by the embodiments of the present invention can further comprise:
[0085] adjusting by the leader node the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein
[0086] the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and
[0087] transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
[0088] The task executing state information includes task executing speed and task accumulation.
[0089] Specifically, the various Worker nodes send the task executing state information and the host state information on the hosts to the leader node;
[0090] the leader node determines tasks in idle states and tasks in stacked states according to the task executing state information, places the tasks in idle states (Idle Tasks) and the tasks in stacked states (Busy Tasks) respectively in an IdleTask queue and a BusyTask queue, performs automatic volume reduction on the Idle Tasks, and performs proactive volume Date Regue/Date Received 2022-07-18 expansion on the Busy Tasks;
[0091] the leader node determines Worker nodes whose host loads are in overload states and Worker nodes whose host loads are in idle states according to the host running state information of the various Worker nodes, and transfers tasks on the Worker nodes whose host loads are in overload states to the Worker nodes whose host loads are in idle states to execute the task.
[0092] In the embodiments of the present invention, by adjusting configuration information of tasks by the leader node according to the host running state information and the task executing state information respectively reported by the plural Worker nodes, it is possible to ensure that tasks on Worker nodes with higher host loads are transferred to other Worker nodes for execution when the host loads of some Worker nodes are relatively higher and machine load of the cluster is uneven, so that load balance of Worker nodes and tasks among clusters is achieved, high availability of Worker nodes is realized, availability of the Flume service is enhanced, and the problems concerning uneven utilization and wastage of resources are solved at the same time.
[0093] As a further preferred embodiment, the method provided by the embodiments of the present invention can further comprise:
[0094] receiving by the leader node a request to go offline of the business through the external interface; and
[0095] writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
[0096] Specifically, the leader node provides an external interface to receive a request to go offline of the business (namely Job), the leader node marks the state of the business going offline as offline, and maintains the going-offline information of the business and going-offline information of the task assigned to the business in the ZooKeeper cluster; when the Worker node that executes the task monitors the going-offline information of the task Date Regue/Date Received 2022-07-18 from the ZooKeeper cluster, it executes a stop command of the task, and stops the Flume service; after all tasks relevant to the business have been stopped for execution, the business completes going offline.
[0097] Fig. 2 is a block diagram illustrating a data flow processing system provided by Embodiment 2 of the present invention, the system comprises a Zookeeper cluster 10, plural Master nodes 20, and plural Worker nodes 30. As shown in Fig. 2, the number of Master nodes 20 can be configured as two, including Master node 21 and Master node 22, when one of the Master nodes serves as the leader node, the other one serves as a standby node, and the number of Worker nodes 30 can be configured as three, including Worker node 31, Worker node 32, and Worker node 33, wherein,
[0098] the Zookeeper cluster is employed for determining one of the plural Master nodes to serve as the leader node;
[0099] the leader node is employed for providing an external interface to receive a request to go online of a business, and assigning a task to the business;
[0100] the leader node is further employed for generating configuration information of the task and writing the same in the Zookeeper cluster according to current state information respectively reported by the plural Worker nodes, wherein the configuration information includes information instructing a Worker node scheduled to execute the task;
and
and
[0101] the Worker node is employed for if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
[0102] Further, the Zookeeper cluster is specifically employed for:
[0103] receiving a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
[0104] the Master nodes being started;
[0105] failure occurring to the current Master node serving as the leader node.
Date Regue/Date Received 2022-07-18
Date Regue/Date Received 2022-07-18
[0106] Moreover, the leader node is specifically employed for:
[0107] determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes; and
[0108] generating configuration information instructing to schedule the task to the target Worker node.
[0109] Furthermore, the leader node is specifically further employed for:
[0110] adjusting the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein
[0111] the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and
[0112] transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
[0113] Moreover, the leader node is specifically further employed for:
[0114] receiving a request to go offline of the business through the external interface; and
[0115] writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
[0116] As should be noted, the data flow processing system provided by the above embodiments is merely exemplarily described by being divided into the aforementioned plural functional modules, whereas in practical application, it is possible to assign these functions to different functional modules to be completed according to requirements, that is to say, the internal structure of the system can be divided into different functional modules to complete the entire or partial functions mentioned above. In addition, the data Date Regue/Date Received 2022-07-18 flow processing system pertains to the same conception as the data flow processing method embodiment¨ see the method embodiment for details of its specific implementation process and advantageous effects, while no repetition is made in this context.
[0117] As can be understood by persons ordinarily skilled in the art, the entire or partial steps that realize the aforementioned embodiments can be completed via hardware, and can also be completed via a program to instruct relevant hardware, the program can be stored in a computer-readable storage medium, and the storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.
[0118] What the above describes is merely directed to preferred embodiments of the present invention, and is not meant to restrict the present invention. Any modification, equivalent substitution, and improvement makeable within the spirit and principle of the present invention shall all be covered by the protection scope of the present invention.
Date Regue/Date Received 2022-07-18
Date Regue/Date Received 2022-07-18
Claims (10)
1. A data flow processing method, characterized in that the method comprises:
determining by a Zookeeper cluster one of plural Master nodes to serve as a leader node;
by the leader node, providing an external interface to receive a request to go online of a business and assigning a task to the business;
generating configuration information of the task and writing the same in the Zookeeper cluster according to host running state information respectively reported by plural Worker nodes, wherein the configuration information includes information instructing a Worker node scheduled to execute the task; and if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
determining by a Zookeeper cluster one of plural Master nodes to serve as a leader node;
by the leader node, providing an external interface to receive a request to go online of a business and assigning a task to the business;
generating configuration information of the task and writing the same in the Zookeeper cluster according to host running state information respectively reported by plural Worker nodes, wherein the configuration information includes information instructing a Worker node scheduled to execute the task; and if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
2. The method according to Claim 1, characterized in that the step of determining by a Zookeeper cluster one of plural Master nodes to serve as a leader node includes:
receiving, by the Zookeeper cluster, a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
the Master nodes being started;
failure occurring to the current Master node serving as the leader node.
receiving, by the Zookeeper cluster, a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
the Master nodes being started;
failure occurring to the current Master node serving as the leader node.
3. The method according to Claim 1, characterized in that the step of generating configuration information of the task according to host running state information respectively reported by plural Worker nodes includes:
determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes;
and Date Regue/Date Received 2022-07-18 generating configuration information instructing to schedule the task to the target Worker node.
determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes;
and Date Regue/Date Received 2022-07-18 generating configuration information instructing to schedule the task to the target Worker node.
4. The method according to Claim 1, characterized in that the method further comprises:
adjusting by the leader node the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
adjusting by the leader node the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
5. The method according to anyone of Claims 1 to 4, characterized in that the method further comprises:
receiving by the leader node a request to go offline of the business through the external interface;
and writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
receiving by the leader node a request to go offline of the business through the external interface;
and writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
6. A data flow processing system, characterized in that the system comprises a Zookeeper cluster, plural Master nodes, and plural Worker nodes, wherein the Zookeeper cluster is employed for determining one of plural Master nodes to serve as a leader node;
the leader node is employed for providing an external interface to receive a request to go online of a business, and assigning a task to the business;
the leader node is further employed for generating configuration information of the task and writing the same in the Zookeeper cluster according to current state information respectively reported by plural Worker nodes, wherein the configuration information includes information Date Regue/Date Received 2022-07-18 instructing a Worker node scheduled to execute the task; and the Worker node is employed for if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
the leader node is employed for providing an external interface to receive a request to go online of a business, and assigning a task to the business;
the leader node is further employed for generating configuration information of the task and writing the same in the Zookeeper cluster according to current state information respectively reported by plural Worker nodes, wherein the configuration information includes information Date Regue/Date Received 2022-07-18 instructing a Worker node scheduled to execute the task; and the Worker node is employed for if the Worker node monitors a task scheduled to it in the Zookeeper cluster, starting a Flume service to execute the task.
7. The system according to Claim 6, characterized in that the Zookeeper cluster is specifically employed for:
receiving a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
the Master nodes being started;
failure occurring to the current Master node serving as the leader node.
receiving a leader node election request initiated by a Master node on the basis of a preset triggering event, and taking the Master node to serve as the leader node if the election is successful, wherein the preset triggering event is one of the following events:
the Master nodes being started;
failure occurring to the current Master node serving as the leader node.
8. The system according to Claim 6, characterized in that the leader node is specifically employed for:
determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes;
and generating configuration information instructing to schedule the task to the target Worker node.
determining a target Worker node with the optimal host running state in the plural Worker nodes according to host running state information respectively reported by the plural Worker nodes;
and generating configuration information instructing to schedule the task to the target Worker node.
9. The system according to Claim 6, characterized in that the leader node is specifically further employed for:
adjusting the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
Date Regue/Date Received 2022-07-18
adjusting the configuration information of the task according to host running state information and task executing state information respectively reported by the plural Worker nodes; wherein the adjusted configuration information of the task instructs to perform a volume reducing process on any task that is in an idle state and to perform a volume expanding process on any task that is in a stacked state; and transferring a task on any Worker node whose host load is in an overload state to a Worker node whose host load is in an idle state to execute the task.
Date Regue/Date Received 2022-07-18
10. The system according to any one of Claims 6 to 9, characterized in that the leader node is specifically further employed for:
receiving a request to go offline of the business through the external interface; and writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
Date Regue/Date Received 2022-07-18
receiving a request to go offline of the business through the external interface; and writing going-offline information of the business and going-offline information of the task assigned to the business in the Zookeeper cluster, so that the Worker node executing the task stops the Flume service.
Date Regue/Date Received 2022-07-18
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910048043.XA CN109857558A (en) | 2019-01-18 | 2019-01-18 | A kind of data flow processing method and system |
| CN201910048043.X | 2019-01-18 | ||
| PCT/CN2019/106779 WO2020147330A1 (en) | 2019-01-18 | 2019-09-19 | Data stream processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA3168286A1 true CA3168286A1 (en) | 2020-07-23 |
Family
ID=66895175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA3168286A Pending CA3168286A1 (en) | 2019-01-18 | 2019-09-19 | Data flow processing method and system |
Country Status (3)
| Country | Link |
|---|---|
| CN (1) | CN109857558A (en) |
| CA (1) | CA3168286A1 (en) |
| WO (1) | WO2020147330A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204418A (en) * | 2021-05-19 | 2021-08-03 | 中国建设银行股份有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113934782A (en) * | 2021-09-22 | 2022-01-14 | 易联众智鼎(厦门)科技有限公司 | DAG model-based data ETL system and using method |
| CN114844799A (en) * | 2022-05-27 | 2022-08-02 | 深信服科技股份有限公司 | Cluster management method and device, host equipment and readable storage medium |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857558A (en) * | 2019-01-18 | 2019-06-07 | 苏宁易购集团股份有限公司 | A kind of data flow processing method and system |
| CN110262882A (en) * | 2019-06-17 | 2019-09-20 | 北京思特奇信息技术股份有限公司 | A kind of distributed communication command scheduling system and method |
| CN111078396B (en) * | 2019-11-22 | 2023-12-19 | 厦门安胜网络科技有限公司 | Distributed data access method and system based on multitasking examples |
| CN111447097A (en) * | 2020-04-20 | 2020-07-24 | 国网甘肃省电力公司信息通信公司 | Cloud platform resource scheduling management method and system |
| CN112052095B (en) * | 2020-09-11 | 2024-04-19 | 成都锋卫科技有限公司 | Distributed high-availability big data mining task scheduling system |
| CN112416550B (en) * | 2020-11-19 | 2024-04-05 | 广州探途网络技术有限公司 | Communication method of crawler scheduling management platform and crawler scheduling management platform system |
| CN113010307B (en) * | 2021-02-25 | 2024-04-05 | 库珀科技集团有限公司 | Multi-chain blockchain browser system and application method thereof |
| CN113364864B (en) * | 2021-06-03 | 2022-09-30 | 上海微盟企业发展有限公司 | Server data synchronization method, system and storage medium |
| CN113342508A (en) * | 2021-07-07 | 2021-09-03 | 湖南快乐阳光互动娱乐传媒有限公司 | Task scheduling method and device |
| CN113254010A (en) * | 2021-07-09 | 2021-08-13 | 广州光点信息科技有限公司 | Visual DAG workflow task scheduling system and operation method thereof |
| CN114124959A (en) * | 2021-12-06 | 2022-03-01 | 天地伟业技术有限公司 | Data processing device of cloud streaming media service and cloud streaming media cluster |
| CN114697328A (en) * | 2022-03-25 | 2022-07-01 | 浪潮云信息技术股份公司 | Method and system for realizing NiFi high-availability cluster mode |
| CN115002122A (en) * | 2022-05-09 | 2022-09-02 | 中盈优创资讯科技有限公司 | Cluster management method and device for data acquisition |
| CN117076257B (en) * | 2023-09-14 | 2024-03-05 | 研华科技(中国)有限公司 | Management method, management server and management system based on server cluster |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030195962A1 (en) * | 2002-04-10 | 2003-10-16 | Satoshi Kikuchi | Load balancing of servers |
| CN102521044B (en) * | 2011-12-30 | 2013-12-25 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
| CN103595651B (en) * | 2013-10-15 | 2017-02-15 | 北京航空航天大学 | Distributed data stream processing method and system |
| KR101858565B1 (en) * | 2016-02-19 | 2018-05-16 | 서영준 | Independent parallel processing method for massive data in distributed platform and system of thereof |
| CN105939389A (en) * | 2016-06-29 | 2016-09-14 | 乐视控股（北京）有限公司 | Load balancing method and device |
| CN106375342A (en) * | 2016-10-21 | 2017-02-01 | 用友网络科技股份有限公司 | Zookeeper-technology-based system cluster method and system |
| CN108241534A (en) * | 2016-12-27 | 2018-07-03 | 阿里巴巴集团控股有限公司 | A kind of task processing, distribution, management, the method calculated and device |
| CN108228393A (en) * | 2017-12-14 | 2018-06-29 | 浙江航天恒嘉数据科技有限公司 | A kind of implementation method of expansible big data High Availabitity |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
| CN109857558A (en) * | 2019-01-18 | 2019-06-07 | 苏宁易购集团股份有限公司 | A kind of data flow processing method and system |
-
2019
- 2019-01-18 CN CN201910048043.XA patent/CN109857558A/en active Pending
- 2019-09-19 CA CA3168286A patent/CA3168286A1/en active Pending
- 2019-09-19 WO PCT/CN2019/106779 patent/WO2020147330A1/en active Application Filing
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204418A (en) * | 2021-05-19 | 2021-08-03 | 中国建设银行股份有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113934782A (en) * | 2021-09-22 | 2022-01-14 | 易联众智鼎(厦门)科技有限公司 | DAG model-based data ETL system and using method |
| CN114844799A (en) * | 2022-05-27 | 2022-08-02 | 深信服科技股份有限公司 | Cluster management method and device, host equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020147330A1 (en) | 2020-07-23 |
| CN109857558A (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA3168286A1 (en) | Data flow processing method and system | |
| US10628273B2 (en) | Node system, server apparatus, scaling control method, and program | |
| CN108845884B (en) | Physical resource allocation method, device, computer equipment and storage medium | |
| EP3567829B1 (en) | Resource management method and apparatus | |
| US9319281B2 (en) | Resource management method, resource management device, and program product | |
| US11640314B2 (en) | Service provision system, resource allocation method, and resource allocation program | |
| WO2017128507A1 (en) | Decentralized resource scheduling method and system | |
| EP3675420B1 (en) | Distributed storage system upgrade management method and device, and distributed storage system | |
| CN111209110B (en) | Task scheduling management method, system and storage medium for realizing load balancing | |
| US7925922B2 (en) | Failover method and system for a computer system having clustering configuration | |
| CN111414241A (en) | Batch data processing method, device and system, computer equipment and computer readable storage medium | |
| CN112910937B (en) | Object scheduling method and device in container cluster, server and container cluster | |
| CN113382077B (en) | Micro-service scheduling method, micro-service scheduling device, computer equipment and storage medium | |
| US20150186181A1 (en) | System and method for supporting flow control in a distributed data grid | |
| CN114064414A (en) | High-availability cluster state monitoring method and system | |
| CN112631764A (en) | Task scheduling method and device, computer equipment and computer readable medium | |
| CN111459639A (en) | Distributed task management platform and method supporting global multi-machine-room deployment | |
| CN111427670A (en) | Task scheduling method and system | |
| KR20140057368A (en) | Live module diagnostic testing | |
| CN113867911A (en) | Task scheduling method and device and micro-service system | |
| WO2021212967A1 (en) | Task scheduling for distributed data processing | |
| CN111200518B (en) | Decentralized HPC computing cluster management method and system based on paxos algorithm | |
| CN113703930A (en) | Task scheduling method, device and system and computer readable storage medium | |
| CN115080199A (en) | Task scheduling method, system, device, storage medium and program product | |
| CN115811549B (en) | Cloud edge resource management scheduling method and system supporting hybrid heterogeneous operation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |