CN106371912B - A kind of resource regulating method and device of streaming computing system - Google Patents
A kind of resource regulating method and device of streaming computing system Download PDFInfo
- Publication number
- CN106371912B CN106371912B CN201510429901.7A CN201510429901A CN106371912B CN 106371912 B CN106371912 B CN 106371912B CN 201510429901 A CN201510429901 A CN 201510429901A CN 106371912 B CN106371912 B CN 106371912B
- Authority
- CN
- China
- Prior art keywords
- thread
- determined
- allocated
- distributed
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The present invention provides the resource regulating methods and device of a kind of streaming computing system, and wherein method includes: S1, the calculating job information for obtaining input；S2, the number of processes determined according to the calculating job information, distribute to working node for the process of corresponding number；S3, be allocated each thread in the thread list determined according to the calculating job information according at least one of following principle: the thread for belonging to same treatment unit is preferentially dispensed in different processes；Thread with correspondence is preferentially dispensed in same process.The present invention can be improved the redundancy ability or system performance of streaming computing system.
Description
[technical field]
The present invention relates to computer application technology, in particular to a kind of resource regulating method of streaming computing system and
Device.
[background technique]
JStorm is the real-time streaming Computational frame with reference to storm, in network I/O, threading model, scheduling of resource, availability
And sustained improvement has been done in stability, it is used by more and more enterprises.JStorm cluster is by a scheduling node and multiple works
Make node composition.Scheduling node is mainly responsible for carry out task schedule, and scheduling result is sent to zookeeper.Working node point
Not Jian Ting zookeeper, obtain distribute to itself task and execute task, main frame is as shown in fig. 1.In each work
Certain Slot resource is owned by node.Each Job (work) can regard Topology (calculating topology) as, wherein wrapping
Containing multi-stage treatment units, the example of each processing unit operation is referred to as Task (task), and a Task is a thread.Operation
The process of Task is known as Worker, and total process number of Job is Worker number.It can in one Slot resource of each working node
Run a Worker.
Current either JStorm or Storm, the scheduling mode used is all Principle of Average Allocation, although improving money
Source utilization rate, but there is no consider redundancy ability and system performance.
[summary of the invention]
In view of this, the present invention provides a kind of method and apparatus of scheduling of resource, in order to improve streaming computing system
Redundancy ability or system performance.
Specific technical solution is as follows:
The present invention provides a kind of resource regulating methods of streaming computing system, this method comprises:
S1, the calculating job information for obtaining input；
S2, the number of processes determined according to the calculating job information, distribute to working node for the process of corresponding number；
S3, by according to it is described calculating job information determine thread list in each thread according in following principle at least
One kind is allocated:
The thread for belonging to same treatment unit is preferentially dispensed in different processes；
Thread with correspondence is preferentially dispensed in same process.
A preferred embodiment according to the present invention determines that number of processes includes: according to the calculating job information
By the sum and working node current residual of the number of processes, thread that include in the calculating job information
Minimum value in Slot sum, as the number of processes determined.
A preferred embodiment according to the present invention, the S2 include:
The process of the corresponding number is allocated one by one, is divided according to current residual Slot number from more to few sequence
Process is not distributed into working node, distributes a process on each Slot.
A preferred embodiment according to the present invention, the thread that same treatment unit is belonged to according to described in are preferentially dispensed on not
The thread in thread list is allocated with the principle in process and includes:
A11, it determines in the process that the S2 is distributed, it is assigned to belong to same treatment with current thread to be allocated
The smallest process of the number of threads of unit；
A12, current thread to be allocated is distributed to one into the A11 determining process.
A preferred embodiment according to the present invention, the A12 include:
If the process that the A11 is determined only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A11 is determined is more than one, A13 is continued to execute；
A13, in the process that the A11 is determined, be determined for compliance with the process of the following conditions: working node where the process
The total number of threads for belonging to same treatment unit with current thread to be allocated that upper all processes have been assigned is minimum；
A14, current thread to be allocated is distributed to one into the A13 determining process.
A preferred embodiment according to the present invention, the A14 include:
If the process that the A13 is determined only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A14 is determined is more than one, A15 is continued to execute；
A15, in the process that the A13 is determined, determination be assigned the smallest process of number of threads；
A16, current thread to be allocated is distributed to one into the A15 determining process.
A preferred embodiment according to the present invention, the A16 include:
If the process that the A15 is determined only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A15 is determined is more than one, A17 is continued to execute；
A17, in the process that the A15 is determined, be determined for compliance with the process of the following conditions: working node where the process
The total number of threads that upper all processes have been assigned is minimum；
A18, current thread to be allocated is distributed to one into the A17 determining process.
A preferred embodiment according to the present invention, according to the thread with correspondence be preferentially dispensed on it is identical into
Principle in journey, which is allocated the thread in thread list, includes:
A21, it determines in the process that the S2 is distributed, it is assigned that there is correspondence with current thread to be allocated
The maximum process of total number of threads；
A22, current thread to be allocated is distributed to one into the A21 determining process.
A preferred embodiment according to the present invention, the A22 include:
If the process that the A21 is determined only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A21 is determined is more than one, A23 is continued to execute；
A23, in the process that the A21 is determined, be determined for compliance with the process of the following conditions: working node where the process
On be assigned have the total number of threads of correspondence maximum with current thread to be allocated；
A24, current thread to be allocated is distributed to one into the A23 determining process.
A preferred embodiment according to the present invention, this method further include:
After one thread of every distribution is to process, judge whether the total number of threads being assigned in the process is more than number of threads
Mean value, if it is, the process is excluded in subsequent distribution；
Wherein the number of threads mean value is asked by the total number of threads determined according to the calculating job information with number of processes
It is determined after mean value.
The present invention also provides a kind of resource scheduling device of streaming computing system, which includes:
Acquiring unit, for obtaining the calculating job information of input；
Process allocation unit, the number of processes for being determined according to the calculating job information, by the process of corresponding number
Distribute to working node；
Thread allocation unit, for will according to it is described calculating job information determine thread list in each thread according to
At least one of lower principle is allocated:
The thread for belonging to same treatment unit is preferentially dispensed in different processes；
Thread with correspondence is preferentially dispensed in same process.
A preferred embodiment according to the present invention, the process allocation unit according to the calculating job information determine into
When number of passes mesh, it is specifically used for: the number of processes for including in the calculating job information, the sum of thread and working node is worked as
Minimum value in preceding remaining Slot sum, as the number of processes determined.
A preferred embodiment according to the present invention, the process allocation unit, be specifically used for: by the corresponding number into
Journey is allocated one by one, process is distributed to working node respectively from more to few sequence according to current residual Slot number, often
A process is distributed on a Slot.
A preferred embodiment according to the present invention, the thread allocation unit are belonging to same treatment unit according to described in
It is specific to execute following behaviour when the principle that thread is preferentially dispensed in different processes is allocated the thread in thread list
Make:
A11, it determines in the process that the process allocation unit is distributed, assigned and current thread category to be allocated
In the smallest process of the number of threads of same treatment unit；
A12, current thread to be allocated is distributed to one into the A11 determining process.
A preferred embodiment according to the present invention, the thread allocation unit are specific to execute when executing operation A12:
If the process determined in the A11 only has one, current thread to be allocated is distributed to the determination
Process out；If the process that the A11 is determined is more than one, A13 is continued to execute；
A13, in the process that the A11 is determined, be determined for compliance with the process of the following conditions: working node where the process
The total number of threads for belonging to same treatment unit with current thread to be allocated that upper all processes have been assigned is minimum；
A14, current thread to be allocated is distributed to one into the A13 determining process.
A preferred embodiment according to the present invention, the thread allocation unit are specific to execute when executing operation A14:
If the process determined in the A13 only has one, current thread to be allocated is distributed to the determination
Process out；If the process that the A14 is determined is more than one, A15 is continued to execute；
A15, in the process that the A13 is determined, determination be assigned the smallest process of number of threads；
A16, current thread to be allocated is distributed to one into the A15 determining process.
A preferred embodiment according to the present invention, the thread allocation unit are specific to execute when executing the A16:
If the process determined in the A15 only has one, current thread to be allocated is distributed to the determination
Process out；If the process that the A156 is determined is more than one, A17 is continued to execute；
A17, in the process that the A15 is determined, be determined for compliance with the process of the following conditions: working node where the process
The total number of threads that upper all processes have been assigned is minimum；
A18, current thread to be allocated is distributed to one into the A17 determining process.
A preferred embodiment according to the present invention, the thread allocation unit is according to the thread with correspondence
It is specific to execute following operation when the principle being preferentially dispensed in same process is allocated the thread in thread list:
A21, it determines in the process that the process allocation unit is distributed, it is assigned to have with current thread to be allocated
There is the maximum process of the total number of threads of correspondence；
A22, current thread to be allocated is distributed to one into the A21 determining process.
A preferred embodiment according to the present invention, the thread allocation unit are specific to execute when executing operation A22:
If only having one in the process that the operation A21 is determined, current thread to be allocated is distributed to this really
The process made；If the process that the A21 is determined is more than one, A23 is continued to execute；
A23, in the process that the A21 is determined, be determined for compliance with the process of the following conditions: working node where the process
On be assigned have the total number of threads of correspondence maximum with current thread to be allocated；
A24, current thread to be allocated is distributed to one into the A23 determining process.
A preferred embodiment according to the present invention, the device further include:
Auxiliary unit is distributed, for judging the process after the thread allocation unit one thread of every distribution is to process
On the total number of threads that has been assigned whether be more than number of threads mean value, if it is, the process is excluded in subsequent distribution；
Wherein the number of threads mean value is asked by the total number of threads determined according to the calculating job information with number of processes
It is determined after mean value.
As can be seen from the above technical solutions, the thread for belonging to same treatment unit is preferentially dispensed on phase in the present invention
With in process, so that the failure of process reduces the influence for calculating work, redundancy ability is improved, or will be with correspondence
Thread is preferentially dispensed in same process, reduces interprocess communication expense caused by cross-thread to the greatest extent, improves system performance.
[Detailed description of the invention]
Fig. 1 is the main frame schematic diagram of JStorm；
Fig. 2 is main method flow chart provided in an embodiment of the present invention；
Fig. 3 is method detailed flow chart provided in an embodiment of the present invention；
Fig. 4 is structure drawing of device provided in an embodiment of the present invention.
[specific embodiment]
To make the objectives, technical solutions, and advantages of the present invention clearer, right in the following with reference to the drawings and specific embodiments
The present invention is described in detail.
Fig. 2 is main method flow chart provided in an embodiment of the present invention, and as shown in Figure 2, this method may include following
Step:
In 201, the Job information of input is obtained.
In the embodiment of the present invention, when needing to run a calculating job in streaming computing system, the usual calculating work
Work is calculated as real-time, Job information can be inputted to streaming computing system, in JStorm or Storm, one calculates work in real time
The logic of work is encapsulated in a Topology object.Worker number, Task, each Task are defined in the Topology
Between execute the oriented cyclic annular figure that relationship is constituted, the sequence that executes between each Task is defined in the oriented cyclic annular figure, usually with
The logical topological structure form of message flow embodies between Task.Worker is to run the process for handling calculating logic, in Worker
Each thread is known as a Task.In addition, usually same kind of Task can be by as a processing unit, processing unit letter
The Task information that breath and processing unit include also is defined in Job information.
It in order to make above-mentioned concept definitely, gives one example herein, it is assumed that there are such a real-time calculating work: logical
It crosses and analysis is carried out to establish the disaggregated model of user demand to search log.This calculates work in real time and is segmented at three
Reason unit: 1) search log is pulled；2) feature extraction and sample classification are carried out to search log；3) feature and sample extracted are utilized
The training of this classification progress disaggregated model.Such as given a definition in Job information: defining 4 Worker, above-mentioned 3 processing are single
Member separately includes 3 Task in each processing unit.It just needs in this way, this calculates work in real time to 4 Worker and 9
A Task is allocated.
In 202, according to load balancing principle, according to the Worker number that the Job information determines, by corresponding number
Worker distributes to working node.
After getting Job information, it is substantially carried out two-part distribution: first distributing Worker to working node, then will
Task is distributed to Worker.This step is the former distribution, using load balancing principle, as far as possible fifty-fifty by Worker points
It is assigned to working node.
In 203, by each Task in the Task list determined according to Job information according at least one of following principle
Be allocated: the Task for belonging to same treatment unit is preferentially dispensed on different Worker；Task with correspondence is excellent
It is first dispensed on identical Worker.
Task defined in Job information constitutes Task list, is allocated respectively to each Task in Task list.Into
When row distribution, the Task for belonging to same treatment unit is preferentially dispensed on different Worker, so that the failure pair of Worker
The influence of Topology reduces, and improves redundancy ability.Task with correspondence is preferentially dispensed on identical Worker, to the greatest extent
Amount reduces the interprocess communication expense between Task, improves system performance.
It should be noted that two principles in above-mentioned steps 203 can all can also be used only with one of those
And preferentially use one of them.
Below by embodiment illustrated in fig. 3 to the above-mentioned two principle of use simultaneously, and preferentially uses and belong to same treatment list
The mode for the principle that the Task of member is preferentially dispensed on different Worker is described in detail.Fig. 3 mentions for the embodiment of the present invention
The method detailed flow chart of confession, as shown in figure 3, the process can with specifically includes the following steps:
In 301, the Job information of input is obtained.
In 302, by Worker number, the sum of Task defined in Job information and working node current residual
Minimum value in Slot sum, as the Worker number determined, it is assumed that the Worker number is determined as N.
Due to can only run a Worker on a Slot, even if Worker number defined in Job information is greater than
The Slot sum of current residual, is only able to run Worker according to the Slot sum of current residual.Additionally, it is also possible to due to
The reason of accidentally defining does not use so much Worker, most so that Worker number is greater than the sum of Task in this case
Mostly only need the Worker of Task sum.It, in this step, can be according to defined in Job information in view of two above reason
Minimum value in the Slot sum of Worker number, the sum of Task and working node current residual, to determine practical distribution
Worker number.
In 303, working node is arranged according to the sequence of remaining Slot number from big to small.
In 304, the Worker of N number of number is allocated one by one, it is most is respectively allocated to current residual Slot number
Working node.
When being allocated to Worker, however, it would be possible to using according to current residual Slot number from more to few sequence
Process is distributed into working node respectively, it is wherein one that the most working node of current residual Slot number is distributed in this step
Kind preferred embodiment.
Above-mentioned steps 303 not necessarily the step of, in embodiments of the present invention, when carrying out Worker distribution, to the greatest extent
Amount is so that the Worker quantity on each working node is average.For example, it is assumed that there are three working nodes: A, B and C, 5
Worker needs to distribute, and the remaining Slot quantity on working node A, B and C is respectively as follows: 3,2,1, then first by one
Worker is distributed to working node A.The remaining Slot quantity on working node A, B and C is respectively as follows: 2,2,1 at this time, from work
An optional working node distributes a Worker in node A and B, it is assumed that distributes second Worker to working node A.This
When working node A, B and C on remaining Slot quantity be respectively as follows: 1,2,1, third Worker is distributed to working node B.This
When working node A, B and C on remaining Slot quantity be respectively as follows: 1,1,1, the 4th Worker is distributed to any one work and is saved
Point, it is assumed that distribution to working node A.The remaining Slot quantity on working node A, B and C is respectively as follows: 0,1,1 at this time, by the 5th
A Worker is distributed to working node B or C.
In 305, Task list is determined according to Job information, according to Task sum and the above-mentioned Worker number N determined
Determine the Task number mean value accommodated on each Worker, it is assumed that be M.
The value that M is determined in this step is retouched in rear extended meeting in order to enable the Task quantity run on Worker is average as far as possible
It states, controls the Task number accommodated on each Worker and be no more than M.The determination of M value can be used and be accommodated on each Worker
The mode that rounds up of Task number mean value.Assuming that Task sum defined in Job information is 12, the Worker number determined
Mesh is 5, then M value takes 3.
In 306, a Task is taken out from Task list and executes following steps respectively as Task to be allocated:
In 307, determine in Worker that step 304 is distributed, it is assigned to belong to phase with currently Task to be allocated
With the smallest Worker of Task number of processing unit.
The Worker that step 304 is distributed can form a Worker list, and Task to be allocated is distributed to this
On some Worker in Worker list.In this step, it is to try to guarantee that the Task of same treatment unit is dispensed on not
With on Worker, to improve redundancy ability.
In 308, judge whether the Worker that 307 determine only has one, if so, executing 309；Otherwise, it executes
310。
In 309, current Task to be allocated is distributed to the Worker determined, step 321 is executed.
In 310, in the Worker determined 307, it is determined for compliance with the Worker of the following conditions: work where the Worker
It is minimum to make the Task sum for belonging to same treatment unit with current Task to be allocated that all Worker on node have been assigned.
The purpose of this step is to try to guarantee that the Task of same treatment unit is dispensed on different operating node, thus
Improve redundancy ability.
In 311, judge whether the Worker that 310 determine only has one, if so, executing 309；Otherwise, it executes
312。
In 312, in the Worker that 310 determine, the smallest Worker of Task number being assigned is determined.
The purpose of this step is to guarantee that Task is averaged distribution on Worker as far as possible.
In 313, judge whether the Worker that 312 determine only has one, if so, executing 309；Otherwise, it executes
314。
In 314, in the Worker that 312 determine, it is determined for compliance with the Worker of the following conditions: work where the Worker
It is minimum to make the Task sum that all Worker have been assigned on node.
The purpose of this step is to guarantee that Task is averaged distribution on working node as far as possible.
In 315, judge whether the Worker that 314 determine only has one, if so, executing 309；Otherwise, it executes
316。
In 316, in the Worker that 314 determine, it is assigned have with current Task to be allocated communicate pass
The maximum Worker of Task sum of system.
The purpose of this step is to guarantee have the Task of correspondence to be dispensed on identical Worker as far as possible, to reduce
Striding course communication, improves system performance.Whether can be from the oriented cyclic annular figure of Job information really with correspondence between Task
It is fixed, if there are correspondences between two Task there are message flow between two Task.
In 317, judge whether the Worker that 316 determine only has one, if so, executing 309；Otherwise, it executes
318。
In 318, in the Worker that 316 determine, it is determined for compliance with the Worker of the following conditions: work where the Worker
That makees to be assigned on node has the Task sum of correspondence maximum with current Task to be allocated.
The purpose of this step is to guarantee have the Task of correspondence to be dispensed on identical working node as far as possible, to subtract
Network communication between few working node, improves system performance.
In 319, judge whether the Worker that 318 determine only has one, if so, executing 309；Otherwise, it executes
320。
In 320, an optional Worker is executed as the Worker determined in the Worker that determines from 318
309。
321: judge whether the Task sum being assigned on the Worker reaches M, if so, step 322 is executed, it is no
Then, 323 are executed.
One Task is distributed to a Worker, if the Task sum being assigned on the Worker reaches M,
Worker is removed from Worker list, is no longer participate in the distribution of subsequent Task, this makes it possible to guarantee that Task is put down as far as possible
It distributes on each Worker.
In 322, which is excluded in subsequent distribution.
In 323, next Task is taken out from Task list as Task to be allocated, goes to and executes step 307, directly
Into Task list, all Task are assigned.
It is the detailed description carried out to method provided by the invention above, device provided by the invention is carried out below detailed
Description.Fig. 4 be streaming computing system provided in an embodiment of the present invention resource scheduling device structure chart, the device can be set in
Host node in streaming computing system, as shown in Figure 4, the apparatus may include: acquiring unit 01,02 and of process allocation unit
Thread allocation unit 03 can also include distribution auxiliary unit 04, and the major function of each component units is as follows:
Acquiring unit 01 is responsible for obtaining the Job information of input.The Job information includes the information for calculating work in real time,
In JStorm or Storm, a logic for calculating work in real time is encapsulated in a Topology object.At this
Worker number, Task are defined in Topology, the oriented cyclic annular figure that relationship is constituted, the oriented ring-type are executed between each Task
The sequence that executes between each Task is defined in figure, is embodied usually in the form of the logical topological structure of message flow between Task.
Process allocation unit 02 is responsible for, according to the Worker number that Job information determines, to correspond to according to load balancing principle
The Worker of number distributes to working node.
Firstly, process allocation unit 02 can will include in Job information when determining Worker number according to Job information
Worker number, Task sum and working node current residual Slot sum in minimum value, as what is determined
Worker number.
In addition, the Worker of corresponding number can be allocated by process allocation unit 02 one by one, according to current residual
Process is distributed to working node respectively from more to few sequence by Slot number, such as is respectively allocated to current residual Slot number
Most working nodes distributes a Worker on each Slot.
Thread allocation unit 03 is responsible for each Task in the Task list that will be determined according to Job information according in following principle
At least one be allocated: the Task for belonging to same treatment unit is preferentially dispensed on different Worker；It is closed with communication
The Task of system is preferentially dispensed on identical Worker.
Task defined in Job information constitutes Task list, is allocated respectively to each Task in Task list.Into
When row distribution, the Task for belonging to same treatment unit is preferentially dispensed on different Worker, so that the failure pair of Worker
The influence of Topology reduces, and improves redundancy ability.Task with correspondence is preferentially dispensed on identical Worker, to the greatest extent
Amount reduces the interprocess communication expense between Task, improves system performance.
Specifically, thread allocation unit 03 is preferentially being dispensed on difference according to the Task for belonging to same treatment unit
When principle on Worker is allocated the Task in Task list, following operation is executed:
A11, it determines in the Worker that process allocation unit 02 is distributed, it is assigned to belong to current Task to be allocated
In the smallest Worker of Task number of same treatment unit.
A12, current Task to be allocated is distributed to one into A11 determining Worker.
Wherein, thread allocation unit 03 is when executing operation A12, if the Worker determined in A11 only has one,
Then current Task to be allocated is distributed to the Worker determined；If the Worker that A11 is determined is more than one, after
It is continuous to execute A13.
A13, in the Worker that A11 is determined, be determined for compliance with the Worker of the following conditions: work section where the Worker
The Task sum for belonging to same treatment unit with current Task to be allocated that all Worker have been assigned on point is minimum.
A14, current Task to be allocated is distributed to one into A13 determining Worker.
Further, thread allocation unit 03 is when executing operation A14, if the Worker determined in A13 is only
There is one, then distributes current Task to be allocated to the Worker determined；If the Worker that A14 is determined is more than one
It is a, then continue to execute A15.
A15, in the Worker that A13 is determined, determination be assigned the smallest Worker of Task number.
A16, current Task to be allocated is distributed to one into A15 determining Worker.
Further, thread allocation unit 03 is when executing A16, if the Worker determined in A15 only has one
It is a, then current Task to be allocated is distributed to the Worker determined；If the Worker that A156 is determined is more than one,
Then continue to execute A17.
A17, in the Worker that A15 is determined, be determined for compliance with the Worker of the following conditions: work section where the Worker
The Task sum that all Worker have been assigned on point is minimum.
A18, current Task to be allocated is distributed to one into A17 determining Worker.
For another principle, thread allocation unit 03 according to have correspondence Task be preferentially dispensed on it is identical
When principle on Worker is allocated the Task in Task list, following operation can be specifically executed:
A21, it determines in the Worker that process allocation unit 02 is distributed, it is assigned to have with current Task to be allocated
There is the maximum Worker of Task sum of correspondence.
A22, current Task to be allocated is distributed to one into A21 determining Worker.
Wherein, thread allocation unit is when executing operation A22, if only having one in the Worker that operation A21 is determined,
Then current Task to be allocated is distributed to the Worker determined；If the Worker that A21 is determined is more than one, after
It is continuous to execute A23.
A23, in the Worker that A21 is determined, be determined for compliance with the Worker of the following conditions: work section where the Worker
What is be assigned on point has the Task sum of correspondence maximum with current Task to be allocated.
A24, current Task to be allocated is distributed to one into A23 determining Worker.Aforesaid operations A18 and A24
In, Task to be allocated can be distributed to determining any of Worker, can also therefrom be selected using other modes
One.
Above-mentioned distribution auxiliary unit 04 can be after one Task of the every distribution of thread allocation unit 03 be to Worker, and judging should
Whether the Task sum being assigned on Worker is more than Task number mean value, if it is, the Worker is excluded in subsequent
Distribution.Wherein Task number mean value determines after being averaged by the Task sum and Worker number determined according to Job information.Point
It can guarantee that Task is fifty-fifty distributed on each Worker as far as possible with auxiliary unit 04.
Above-mentioned two principle can use simultaneously, and be preferentially preferentially dispensed on using the Task for belonging to same treatment unit
Principle on different Worker.In this case, thread allocation unit 03 is after executing operation A18, if it is determined that goes out
Worker number is more than one, then continuing to execute operation A21, operates the Worker for determining in A21 and determining in operation A17 at this time
In, the assigned maximum Work of Task sum with current Task to be allocated with correspondence.
After host node in streaming computing system completes the distribution of Worker and Task by above-mentioned apparatus, by allocation result
It send to zookeeper, working node monitors zookeeper, obtains the worker and task that distribute to itself so as to subsequent execution.
It should be noted that the above embodiment of the present invention is only described by taking JStorm and Storm as an example, for other classes
Like the streaming computing system of framework, method and apparatus provided in an embodiment of the present invention are equally applicable, it is only possible to be related to counting in real time
The difference nominally such as work, process, thread is calculated, core concept is similar with process is executed.
In several embodiments provided by the present invention, it should be understood that disclosed device and method can pass through it
Its mode is realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the unit, only
Only a kind of logical function partition, there may be another division manner in actual implementation.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
It is each that equipment (can be personal computer, server or the network equipment etc.) or processor (processor) execute the present invention
The part steps of embodiment the method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (Read-
Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. it is various
It can store the medium of program code.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the present invention.
Claims (18)
1. a kind of resource regulating method of streaming computing system, which is characterized in that this method comprises:
S1, the calculating job information for obtaining input；
S2, the number of processes determined according to the calculating job information, distribute to working node for the process of corresponding number；
S3, by according to it is described calculating job information determine thread list in each thread according at least one of following principle
It is allocated:
The thread for belonging to same treatment unit is preferentially dispensed in different processes；
Thread with correspondence is preferentially dispensed in same process；
The thread for belonging to same treatment unit according to described in is preferentially dispensed on principle in different processes in thread list
Thread, which is allocated, includes:
A11, it determines in the process that the S2 is distributed, it is assigned to belong to same treatment unit with current thread to be allocated
The smallest process of number of threads；
A12, current thread to be allocated is distributed to one into the A11 determining process.
2. the method according to claim 1, wherein determining number of processes packet according to the calculating job information
It includes:
Number of processes, the sum of thread and the resource block of working node current residual that will include in the calculating job information
Minimum value in Slot sum, as the number of processes determined.
3. method according to claim 1 or 2, which is characterized in that the S2 includes:
The process of the corresponding number is allocated one by one, respectively will according to current residual Slot number from more to few sequence
Process distributes to working node, distributes a process on each Slot.
4. the method according to claim 1, wherein the A12 includes:
If the process that the A11 is determined only has one, current thread to be allocated is distributed to the process determined；
If the process that the A11 is determined is more than one, A13 is continued to execute；
A13, in the process that the A11 is determined, be determined for compliance with the process of the following conditions: institute on working node where the process
The total number of threads for belonging to same treatment unit with current thread to be allocated for having process to be assigned is minimum；
A14, current thread to be allocated is distributed to one into the A13 determining process.
5. according to the method described in claim 4, it is characterized in that, the A14 includes:
If the process that the A13 is determined only has one, current thread to be allocated is distributed to the process determined；
If the process that the A13 is determined is more than one, A15 is continued to execute；
A15, in the process that the A13 is determined, determination be assigned the smallest process of number of threads；
A16, current thread to be allocated is distributed to one into the A15 determining process.
6. according to the method described in claim 5, it is characterized in that, the A16 includes:
If the process that the A15 is determined only has one, current thread to be allocated is distributed to the process determined；
If the process that the A15 is determined is more than one, A17 is continued to execute；
A17, in the process that the A15 is determined, be determined for compliance with the process of the following conditions: institute on working node where the process
The total number of threads for having process to be assigned is minimum；
A18, current thread to be allocated is distributed to one into the A17 determining process.
7. the method according to claim 1, wherein being preferentially assigned according to the thread with correspondence
Principle in same process, which is allocated the thread in thread list, includes:
A21, it determines in the process that the S2 is distributed, the assigned line with current thread to be allocated with correspondence
The maximum process of journey sum；
A22, current thread to be allocated is distributed to one into the A21 determining process.
8. the method according to the description of claim 7 is characterized in that the A22 includes:
If the process that the A21 is determined only has one, current thread to be allocated is distributed to the process determined；
If the process that the A21 is determined is more than one, A23 is continued to execute；
A23, in the process that the A21 is determined, be determined for compliance with the process of the following conditions: on working node where the process
Assigned has the total number of threads of correspondence maximum with current thread to be allocated；
A24, current thread to be allocated is distributed to one into the A23 determining process.
9. according to claim 1, method described in 4 to 8 any claims, which is characterized in that this method further include:
After one thread of every distribution is to process, judge whether the total number of threads being assigned in the process is more than that number of threads is equal
Value, if it is, the process is excluded in subsequent distribution；
Wherein the number of threads mean value is averaged by the total number of threads determined according to the calculating job information with number of processes
After determine.
10. a kind of resource scheduling device of streaming computing system, which is characterized in that the device includes:
Acquiring unit, for obtaining the calculating job information of input；
Process allocation unit, the number of processes for determining according to the calculating job information, the process of corresponding number is distributed
To working node；
Thread allocation unit, each thread in thread list for will determine according to the calculating job information is according to following original
At least one of then it is allocated:
The thread for belonging to same treatment unit is preferentially dispensed in different processes；
Thread with correspondence is preferentially dispensed in same process；
The thread allocation unit is preferentially dispensed in different processes in the thread for belonging to same treatment unit according to described in
It is specific to execute following operation when principle is allocated the thread in thread list:
A11, it determines in the process that the process allocation unit is distributed, it is assigned to belong to phase with current thread to be allocated
With the smallest process of number of threads of processing unit；
A12, current thread to be allocated is distributed to one into the A11 determining process.
11. device according to claim 10, which is characterized in that the process allocation unit is according to the calculating work
When information determines number of processes, be specifically used for: by it is described calculating job information in include number of processes, thread sum and
Minimum value in the Slot sum of working node current residual, as the number of processes determined.
12. by device described in claim 10 or 11, which is characterized in that the process allocation unit is specifically used for: will be described
The process of corresponding number is allocated one by one, respectively distributes to process from more to few sequence according to current residual Slot number
Working node distributes a process on each Slot.
13. device according to claim 10, which is characterized in that the thread allocation unit is when executing operation A12, tool
Body executes:
If the process determined in the A11 only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A11 is determined is more than one, A13 is continued to execute；
A13, in the process that the A11 is determined, be determined for compliance with the process of the following conditions: institute on working node where the process
The total number of threads for belonging to same treatment unit with current thread to be allocated for having process to be assigned is minimum；
A14, current thread to be allocated is distributed to one into the A13 determining process.
14. device according to claim 13, which is characterized in that the thread allocation unit is when executing operation A14, tool
Body executes:
If the process determined in the A13 only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A13 is determined is more than one, A15 is continued to execute；
A15, in the process that the A13 is determined, determination be assigned the smallest process of number of threads；
A16, current thread to be allocated is distributed to one into the A15 determining process.
15. device according to claim 14, which is characterized in that the thread allocation unit is when executing the A16, tool
Body executes:
If the process determined in the A15 only has one, current thread to be allocated is distributed to this and is determined
Process；If the process that the A15 is determined is more than one, A17 is continued to execute；
A17, in the process that the A15 is determined, be determined for compliance with the process of the following conditions: institute on working node where the process
The total number of threads for having process to be assigned is minimum；
A18, current thread to be allocated is distributed to one into the A17 determining process.
16. device according to claim 10, which is characterized in that the thread allocation unit has communication according to described
When the principle that the thread of relationship is preferentially dispensed in same process is allocated the thread in thread list, it is specific execute with
Lower operation:
A21, it determines in the process that the process allocation unit is distributed, assigned and current thread to be allocated has logical
The maximum process of the total number of threads of gateway system；
A22, current thread to be allocated is distributed to one into the A21 determining process.
17. device according to claim 16, which is characterized in that the thread allocation unit is when executing operation A22, tool
Body executes:
If only having one in the process that the operation A21 is determined, current thread to be allocated is distributed to this and is determined
Process；If the process that the A21 is determined is more than one, A23 is continued to execute；
A23, in the process that the A21 is determined, be determined for compliance with the process of the following conditions: on working node where the process
Assigned has the total number of threads of correspondence maximum with current thread to be allocated；
A24, current thread to be allocated is distributed to one into the A23 determining process.
18. device described in 0,13 to 17 any claims according to claim 1, which is characterized in that the device further include:
Auxiliary unit is distributed, for judging in the process after the thread allocation unit one thread of every distribution is to process
Whether assigned total number of threads is more than number of threads mean value, if it is, the process is excluded in subsequent distribution；
Wherein the number of threads mean value is averaged by the total number of threads determined according to the calculating job information with number of processes
After determine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510429901.7A CN106371912B (en) | 2015-07-21 | 2015-07-21 | A kind of resource regulating method and device of streaming computing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510429901.7A CN106371912B (en) | 2015-07-21 | 2015-07-21 | A kind of resource regulating method and device of streaming computing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106371912A CN106371912A (en) | 2017-02-01 |
| CN106371912B true CN106371912B (en) | 2019-11-26 |
Family
ID=57879758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510429901.7A Active CN106371912B (en) | 2015-07-21 | 2015-07-21 | A kind of resource regulating method and device of streaming computing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106371912B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107528924B (en) * | 2017-10-09 | 2021-06-25 | 郑州云海信息技术有限公司 | Distributed cluster metadata service deployment method and system |
| CN111290744B (en) * | 2020-01-22 | 2023-07-21 | 北京百度网讯科技有限公司 | Stream type computing job processing method, stream type computing system and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1289962A (en) * | 1999-09-23 | 2001-04-04 | 国际商业机器公司 | Establishment of multiple process spanned communication programme in multiple linear equation running environment |
| CN101556545A (en) * | 2009-05-22 | 2009-10-14 | 北京星网锐捷网络技术有限公司 | Method for realizing process support, device and multithreading system |
| CN103473287A (en) * | 2013-08-30 | 2013-12-25 | 中国科学院信息工程研究所 | Method and system for automatically distributing, running and updating executable programs |
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104375882A (en) * | 2014-11-21 | 2015-02-25 | 北京应用物理与计算数学研究所 | Multistage nested data drive calculation method matched with high-performance computer structure |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9338061B2 (en) * | 2012-04-26 | 2016-05-10 | Hewlett Packard Enterprise Development Lp | Open station as a stream analysis operator container |
-
2015
- 2015-07-21 CN CN201510429901.7A patent/CN106371912B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1289962A (en) * | 1999-09-23 | 2001-04-04 | 国际商业机器公司 | Establishment of multiple process spanned communication programme in multiple linear equation running environment |
| CN101556545A (en) * | 2009-05-22 | 2009-10-14 | 北京星网锐捷网络技术有限公司 | Method for realizing process support, device and multithreading system |
| CN103473287A (en) * | 2013-08-30 | 2013-12-25 | 中国科学院信息工程研究所 | Method and system for automatically distributing, running and updating executable programs |
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104375882A (en) * | 2014-11-21 | 2015-02-25 | 北京应用物理与计算数学研究所 | Multistage nested data drive calculation method matched with high-performance computer structure |
Non-Patent Citations (1)
| Title |
|---|
| "Adaptive Online Scheduling in Storm";Leonardo Aniello等;《ACM》;20130703;第207-210页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106371912A (en) | 2017-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020024442A1 (en) | Resource allocation method and apparatus, computer device and computer-readable storage medium | |
| US20210006505A1 (en) | A bursty traffic allocation method, device and proxy server | |
| CN105912399B (en) | Task processing method, device and system | |
| US20170300367A1 (en) | Streaming Graph Optimization Method and Apparatus | |
| CN111625331B (en) | Task scheduling method, device, platform, server and storage medium | |
| CN110333937A (en) | Task distribution method, device, computer equipment and storage medium | |
| US9130844B1 (en) | Systems and methods for harvesting excess compute capacity across domains | |
| CN108270805B (en) | Resource allocation method and device for data processing | |
| CN110968424B (en) | Resource scheduling method, device and storage medium based on K8s | |
| WO2022188578A1 (en) | Method and system for multiple services to share same gpu, and device and medium | |
| CN108123980A (en) | A kind of resource regulating method and system | |
| CN113032102B (en) | Resource rescheduling method, device, equipment and medium | |
| CN112015549B (en) | Method and system for selectively preempting scheduling nodes based on server cluster | |
| CN110275760A (en) | Process based on fictitious host computer processor hangs up method and its relevant device | |
| CN106371912B (en) | A kind of resource regulating method and device of streaming computing system | |
| CN109032769B (en) | Container-based continuous integrated CI (CI) task processing method and device | |
| Tammaro et al. | Dynamic resource allocation in cloud environment under time-variant job requests | |
| Keerthika et al. | A hybrid scheduling algorithm with load balancing for computational grid | |
| CN109614210B (en) | Storm big data energy-saving scheduling method based on energy consumption perception | |
| Fan et al. | Adaptive task scheduling in storm | |
| CN111858458B (en) | Method, device, system, equipment and medium for adjusting interconnection channel | |
| CN111857990B (en) | Method and system for enhancing YARN long-type service scheduling | |
| CN110413393B (en) | Cluster resource management method and device, computer cluster and readable storage medium | |
| CN105138391B (en) | The multitasking virtual machine distribution method of cloud system justice is distributed towards wide area | |
| Zhang et al. | Self-configuration of the number of concurrently running MapReduce jobs in a hadoop cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |