CN111367653A - Stream computing task management method - Google Patents

Stream computing task management method Download PDF

Info

Publication number
CN111367653A
CN111367653A CN202010060960.2A CN202010060960A CN111367653A CN 111367653 A CN111367653 A CN 111367653A CN 202010060960 A CN202010060960 A CN 202010060960A CN 111367653 A CN111367653 A CN 111367653A
Authority
CN
China
Prior art keywords
node
resource
task
resources
threshold
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202010060960.2A
Other languages
Chinese (zh)
Inventor
吕太之
张娟
周彩根
刘永玲
张军
乔大雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Maritime Institute
Original Assignee
Jiangsu Maritime Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Maritime Institute filed Critical Jiangsu Maritime Institute
Priority to CN202010060960.2A priority Critical patent/CN111367653A/en
Publication of CN111367653A publication Critical patent/CN111367653A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention particularly relates to a stream calculation task management method, which comprises the following steps: the main control node receives the streaming computing task and distributes the streaming computing task to each working node of a target streaming computing center server cluster or a target streaming computing unit server cluster; the working node performs stream computing according to the distributed tasks; the coordination node judges whether the streaming calculation task of the working node is larger than a preset maximum bearable threshold range or not in the process that each working node executes the streaming calculation task; and if so, migrating the task of the working node to the working node capable of accommodating the task. The invention judges whether the resources of the node need to be migrated or not and whether the resource residue of the migrated node meets the migration requirement or not by establishing a resource occupation model and a resource residue model for the resources of the node calculated by the flow.

Description

Stream computing task management method
Technical Field
The invention relates to the technical field of stream computing, in particular to a stream computing task management method.
Background
With the explosive growth of global information volume, the big data era has also come. Stream computing is an important class of computing modes for large data. Different from the traditional batch processing calculation based on data scale determination, the flow calculation has the characteristics of infinite data scale, continuous, rapid and unordered data arrival, data volatility, data processing diversification and the like. The new characteristics of the streaming big data are oriented, and how to construct a big data streaming computing system with low delay, high throughput and high reliability is still an open technical problem. The task is a basic unit executed by data processing logic in the streaming computing system and is also a basic unit for scheduling resources of the streaming computing system. Task management is one of core functions of a streaming computing system, and performs resource scheduling and full-life-cycle management on tasks included in streaming applications, including key technologies such as task resource scheduling, data distribution, task fault tolerance and the like. Good task management design provides guarantees for the high efficiency and high reliability of stream computing systems.
Disclosure of Invention
1. The technical problem to be solved is as follows:
in view of the above technical problems, the present invention provides a method for managing a flow calculation task, which performs resource scheduling on a node performing flow calculation.
2. The technical scheme is as follows:
a stream computation task management method is characterized in that: the method comprises the following steps:
the main control node receives the streaming computing task and distributes the streaming computing task to each working node of a target streaming computing center server cluster or a target streaming computing unit server cluster; the working node performs stream computing according to the distributed tasks;
the coordination node judges whether the streaming calculation task of the working node is larger than a preset maximum bearable threshold range or not in the process that each working node executes the streaming calculation task; and if so, migrating the task of the working node to the working node capable of accommodating the task.
Further, the threshold range for the maximum bearer capability of the node includes: CPU occupancy rate threshold, memory occupancy rate threshold, and network bandwidth occupancy rate threshold.
Further, the judgment of the maximum bearable threshold value range is carried out according to a resource occupation model of the working node; judging the working nodes capable of receiving migration according to the resource residual model of the working nodes;
the resource occupation model specifically comprises: each node N ═ N of flow type computing center server cluster1，n2，…，n|N|The hardware resources are divided into CPU resources, memory resources and network bandwidth resources, i.e. hardware resource R ═ { R ═ R }C，RM， RB}; wherein the CPU resource is
Figure RE-GDA0002484514270000011
The memory resource is
Figure RE-GDA0002484514270000012
Figure RE-GDA0002484514270000021
Network bandwidth resources of
Figure RE-GDA0002484514270000022
Wherein task eijOperating at node nkThe occupied CPU resource is
Figure RE-GDA0002484514270000023
Occupied memory resource is
Figure RE-GDA0002484514270000024
Occupied network bandwidth resources are
Figure RE-GDA0002484514270000025
The resource occupancy model is:
Figure RE-GDA0002484514270000026
(1) in the formula, alpha is a preset occupancy threshold of the CPU, β is a preset memory occupancy threshold, and gamma is a network bandwidth occupancy threshold.
The resource residual model specifically comprises:
Figure RE-GDA0002484514270000027
(2) in the formula:
Figure RE-GDA0002484514270000028
are respectively a node nkThe remaining CPU, memory and network bandwidth resources.
Further, the criterion for judging whether the streaming computation task of the working node is larger than the preset maximum bearable threshold range is whether the criterion conforms to the formula (1); the threshold range of the work node receiving the migration task is determined according to the formula (2) that the remaining resources are larger than the resources required by the migration task.
Further, the migration of the task includes: and if the resource occupancy of the plurality of nodes exceeds the threshold, respectively migrating different nodes to different nodes which do not exceed the threshold. If more than 1 resource on the same node exceeds the threshold, setting the memory resource to be the highest priority, the CPU to be the second priority and the network bandwidth resource to be the third priority; and more than 1 resource on the same node exceeds the threshold value and is migrated to the node meeting the condition according to the priority of the resource.
Further, the system also comprises a control node; the control node is the state of the system control and the system accessed by the universal user through the browser.
3. Has the advantages that:
the invention judges whether the resources of the node need to be migrated or not and whether the resource residue of the migrated node meets the migration requirement or not by establishing a resource occupation model and a resource residue model for the resources of the node calculated by the flow.
Drawings
FIG. 1 is a schematic diagram of the present invention.
Detailed Description
A stream computation task management method is characterized in that: as shown in fig. 1, the method comprises the following steps:
the main control node receives the streaming computing task and distributes the streaming computing task to each working node of a target streaming computing center server cluster or a target streaming computing unit server cluster; the working node performs stream computing according to the distributed tasks;
the coordination node judges whether the streaming calculation task of the working node is larger than a preset maximum bearable threshold range or not in the process that each working node executes the streaming calculation task; and if so, migrating the task of the working node to the working node capable of accommodating the task.
Further, the threshold range for the maximum bearer capability of the node includes: CPU occupancy rate threshold, memory occupancy rate threshold, and network bandwidth occupancy rate threshold.
Further, the judgment of the maximum bearable threshold value range is carried out according to a resource occupation model of the working node; judging the working nodes capable of receiving migration according to the resource residual model of the working nodes;
the resource occupation model specifically comprises: each node N ═ N of flow type computing center server cluster1，n2，…，n|N|The hardware resources are divided into CPU resources, memory resources and network bandwidth resources, i.e. hardware resource R ═ { R ═ R }C，RM， RB}; wherein the CPU resource is
Figure RE-GDA0002484514270000031
The memory resource is
Figure RE-GDA0002484514270000032
Figure RE-GDA0002484514270000033
Network bandwidth resources of
Figure RE-GDA0002484514270000034
Wherein task eijOperating at node nkThe occupied CPU resource is
Figure RE-GDA0002484514270000035
Occupied memory resource is
Figure RE-GDA0002484514270000036
Occupied network bandwidth resources are
Figure RE-GDA0002484514270000037
The resource occupancy model is:
Figure RE-GDA0002484514270000038
(1) in the formula, alpha is a preset occupancy threshold of the CPU, β is a preset memory occupancy threshold, and gamma is a network bandwidth occupancy threshold.
The resource residual model specifically comprises:
Figure RE-GDA0002484514270000039
(2) in the formula:
Figure RE-GDA00024845142700000310
are respectively a node nkThe remaining CPU, memory and network bandwidth resources.
Further, the criterion for judging whether the streaming computation task of the working node is larger than the preset maximum bearable threshold range is whether the criterion conforms to the formula (1); the threshold range of the work node receiving the migration task is determined according to the formula (2) that the remaining resources are larger than the resources required by the migration task.
Further, the migration of the task includes: and if the resource occupancy of the plurality of nodes exceeds the threshold, respectively migrating different nodes to different nodes which do not exceed the threshold. If more than 1 resource on the same node exceeds the threshold, setting the memory resource to be the highest priority, the CPU to be the second priority and the network bandwidth resource to be the third priority; and more than 1 resource on the same node exceeds the threshold value and is migrated to the node meeting the condition according to the priority of the resource.
Further, the system also comprises a control node; the control node is the state of the system control and the system accessed by the universal user through the browser.
Although the present invention has been described with reference to the preferred embodiments, it should be understood that various changes and modifications can be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (6)

1. A stream computation task management method is characterized in that: the method comprises the following steps:
the main control node receives the streaming computing task and distributes the streaming computing task to each working node of a target streaming computing center server cluster or a target streaming computing unit server cluster; the working node performs stream computing according to the distributed tasks;
the coordination node judges whether the streaming calculation task of the working node is larger than a preset maximum bearable threshold range or not in the process that each working node executes the streaming calculation task; and if so, migrating the task of the working node to the working node capable of accommodating the task.
2. A stream computation task management method according to claim 1, characterized in that: the maximum threshold range capable of being carried by the working node comprises: CPU occupancy rate threshold, memory occupancy rate threshold, and network bandwidth occupancy rate threshold.
3. A stream computation task management method according to claim 1, characterized in that: judging the maximum bearable threshold range according to a resource occupation model of the working node; judging the working nodes capable of receiving migration according to the resource residual model of the working nodes;
the resource occupation model specifically comprises: each node N ═ N of flow type computing center server cluster1，n2，…，n|N|The hardware resources are divided into CPU resources, memory resources and network bandwidth resources, i.e. hardware resource R ═ { R ═ R }C，RM，RB}；
Wherein the CPU resource is
Figure FDA0002374453260000011
The memory resource is
Figure FDA0002374453260000012
Figure FDA0002374453260000013
Network bandwidth resources of
Figure FDA0002374453260000014
Wherein task eijOperating at node nkThe occupied CPU resource is
Figure FDA0002374453260000015
Occupied memory resource is
Figure FDA0002374453260000016
Occupied network bandwidth resources are
Figure FDA0002374453260000017
The resource occupancy model is:
Figure FDA0002374453260000018
Figure FDA0002374453260000019
Figure FDA00023744532600000110
(1) wherein, alpha is a preset occupancy rate threshold of the CPU, β is a preset memory occupancy rate threshold, and gamma is a network bandwidth occupancy rate threshold;
the resource residual model specifically comprises:
Figure FDA00023744532600000111
Figure FDA00023744532600000112
Figure FDA00023744532600000113
(2) in the formula:
Figure FDA00023744532600000114
are respectively a node nkThe remaining CPU, memory and network bandwidth resources.
4. A stream computation task management method according to claim 3, characterized in that: judging whether the flow type calculation task of the working node is larger than a preset maximum bearable threshold range according to the judgment result of whether the flow type calculation task conforms to the formula (1); the threshold range of the work node receiving the migration task is determined according to the formula (2) that the remaining resources are larger than the resources required by the migration task.
5. A stream computation task management method according to claim 1, characterized in that: the migration of the task comprises:
if the resource occupancy of the plurality of nodes exceeds the threshold, respectively migrating different nodes to different nodes which do not exceed the threshold;
if more than 1 resource on the same node exceeds the threshold, setting the memory resource to be the highest priority, the CPU to be the second priority and the network bandwidth resource to be the third priority; and more than 1 resource on the same node exceeds the threshold value and is migrated to the node meeting the condition according to the priority of the resource.
6. A stream computation task management method according to claim 1, characterized in that: the system also comprises a control node; the control node is the state of the system control and the system accessed by the universal user through the browser.
CN202010060960.2A 2020-01-19 2020-01-19 Stream computing task management method Withdrawn CN111367653A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010060960.2A CN111367653A (en) 2020-01-19 2020-01-19 Stream computing task management method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010060960.2A CN111367653A (en) 2020-01-19 2020-01-19 Stream computing task management method

Publications (1)

Publication Number Publication Date
CN111367653A true CN111367653A (en) 2020-07-03

Family

ID=71206148

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010060960.2A Withdrawn CN111367653A (en) 2020-01-19 2020-01-19 Stream computing task management method

Country Status (1)

Country Link
CN (1) CN111367653A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116055499A (en) * 2023-04-03 2023-05-02 成都四方伟业软件股份有限公司 Method, equipment and medium for intelligently scheduling cluster tasks based on redis

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116055499A (en) * 2023-04-03 2023-05-02 成都四方伟业软件股份有限公司 Method, equipment and medium for intelligently scheduling cluster tasks based on redis

Similar Documents

Publication Publication Date Title
Tantawi et al. Optimal static load balancing in distributed computer systems
CN108089921B (en) Server for cloud big data operation architecture and operation resource optimization method thereof
US7752628B2 (en) Method and apparatus for reassigning objects to processing units
CN109617826B (en) Storm dynamic load balancing method based on cuckoo search
WO2015196911A1 (en) Data mining method and node
Feng et al. Optimal state-free, size-aware dispatching for heterogeneous M/G/-type systems
CN103763378A (en) Task processing method and system and nodes based on distributive type calculation system
Henderson et al. Performance evaluation and policy selection in multiclass networks
CN113033800A (en) Distributed deep learning method and device, parameter server and main working node
CN102314336A (en) Data processing method and system
US7539995B2 (en) Method and apparatus for managing an event processing system
Wang et al. Task scheduling for MapReduce in heterogeneous networks
Anton et al. A survey of stability results for redundancy systems
CN111367653A (en) Stream computing task management method
CN114356531A (en) Edge calculation task classification scheduling method based on K-means clustering and queuing theory
Nazarathy et al. Near optimal control of queueing networks over a finite time horizon
CN111580950A (en) Self-adaptive feedback resource scheduling method for improving cloud reliability
WO2017113865A1 (en) Method and device for big data increment calculation
CN114201306B (en) Multi-dimensional geographic space entity distribution method and system based on load balancing technology
CN116954905A (en) Task scheduling and migration method for large Flink data
CN111258730A (en) Task scheduling method based on competition conditions
Borst et al. Task allocation in a multi-server system
Kirsal Analytical modelling and optimization analysis of large-scale communication systems and networks with repairmen policy
CN114938376A (en) Industrial Internet of things based on priority processing data and control method thereof
CN111858051B (en) Real-time dynamic scheduling method, system and medium suitable for edge computing environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20200703

WW01 Invention patent application withdrawn after publication