CN105049268B - Distributed computing resource distribution system and task processing method - Google Patents

Distributed computing resource distribution system and task processing method Download PDF

Info

Publication number
CN105049268B
CN105049268B CN201510544350.9A CN201510544350A CN105049268B CN 105049268 B CN105049268 B CN 105049268B CN 201510544350 A CN201510544350 A CN 201510544350A CN 105049268 B CN105049268 B CN 105049268B
Authority
CN
China
Prior art keywords
task
computing unit
server
node
video image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201510544350.9A
Other languages
Chinese (zh)
Other versions
CN105049268A (en
Inventor
周光明
张丛喆
万定锐
郭军
王朝昕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Netposa Technologies Ltd
Original Assignee
Netposa Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Netposa Technologies Ltd filed Critical Netposa Technologies Ltd
Priority to CN201510544350.9A priority Critical patent/CN105049268B/en
Publication of CN105049268A publication Critical patent/CN105049268A/en
Application granted granted Critical
Publication of CN105049268B publication Critical patent/CN105049268B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The present invention provides a kind of distributed computing resource distribution system and task processing methods, the system, it include: the video image task that central server is used to receive user's transmission, according to the type of video image task, and task quantity and loading condition that multiple node servers are current, video image task is distributed into corresponding node server by computing unit center-side；Node server is for handling video image task by the corresponding computing unit node side of video image task.Distributed computing resource distribution system provided by the invention and task processing method can be used the different types of video image task of multiple server process an of server cluster, improve the utilization rate of server cluster.

Description

Distributed computing resource distribution system and task processing method
Technical field
The present invention relates to multimedia fields, in particular to a kind of distributed computing resource distribution system and appoint Business processing method.
Background technique
Currently, in daily life, based on video using more and more extensive, the mode handled video is also more next It is more, such as image conversion, the rotation of picture in video, comparison of realtime graphic etc. in video of video, for video difference Processing mode, in order to accelerate the processing speed to video or image, can using different calculating modes to video at Reason, such as: for video conversion process when, since the data volume of processing is larger, so can generally use parallel computation mode (such as MapReduce) post-processes video piecemeal；And when being directed to the comparison of realtime graphic in video, due to needing in a short time The picture material in video image is determined, so carrying out the comparison of mass picture using calculation in memory.So will portion Different server clusters is affixed one's name to realize different calculations.
It, will be video text when needing the image to video to convert in the existing treatment process to video Part is assigned in the server cluster with parallel computation mode and is handled, when needing to compare to realtime graphic in video When, video image will be assigned in the server cluster with calculation in memory and be handled.
It is existing video is handled as unit of server cluster in the way of in, a server cluster can only be realized A kind of video processing mode must just dispose many server clusters of quantity just when the processing mode to video has very much It can satisfy the process demand of video, so the construction cost of server cluster is just considerably increased, moreover, each server set The utilization rate of group is different, and the server resource in the low server cluster of utilization rate cannot be by other video processing modes It is used, causes the waste of resource.
Summary of the invention
The purpose of the present invention is to provide a kind of distributed computing resource distribution system and task processing methods, can be used The different types of video image task of multiple server process of one server cluster, improves the utilization rate of server cluster.
In a first aspect, the embodiment of the invention provides a kind of distributed computing resource distribution systems, comprising: central server With multiple node servers, wherein the computing unit center-side for different task type is provided in the central server, Each node server is respectively arranged with computing unit node side corresponding with the computing unit center-side, the calculating Unit center end and the corresponding computing unit node side form a computing unit；
The central server is used to receive the video image task of user's transmission, according to the class of the video image task Type and the current task quantity and loading condition of the multiple node server, pass through calculating for the video image task Distribute to corresponding node server in unit center end；
The node server is used for through the corresponding computing unit node side of the video image task to the video Image task is handled.
With reference to first aspect, the embodiment of the invention provides the first possible embodiments of first aspect, wherein institute Stating central server includes: task quantity determining module, for having divided according to node server each in current server cluster With task quantity, determination has distributed task quantity least node server；First choice module, for selecting task quantity most Few node server is as the corresponding node server of the video image task；Second selecting module, for multiple when having When the task quantity of distribution of node server is minimum, the section that resource utilization is minimum in the multiple node server is selected Point server comprises at least one of the following clothes as the corresponding node server of the video image task, the resource utilization The utilization rate of business device hardware resource: central processing unit, memory and network bandwidth；
The node server includes: information reporting module, current to central server feedback for periodicity Resource utilization.
With reference to first aspect, the embodiment of the invention provides second of possible embodiments of first aspect, wherein institute Stating central server includes: task adding module, for searching the task of the corresponding node server of the video image task List adds the video image task in the task quantity of distribution of the task list；Task removing module, for working as When receiving the completed information of the video image task that the computing unit node side is sent, by the task list The deletion of video image task described in task quantity is distributed；
The node server includes: task feedback module, is used for when the video image task is completed, by described The corresponding computing unit node side of video image task sends the completed information of task, the task to the central server The mark of node server where completed information carries corresponding computing unit.
With reference to first aspect, the embodiment of the invention provides the third possible embodiment of first aspect, it is described in Central server includes: instruction acquisition module, and for obtaining computing unit addition instruction, the computing unit addition instruction is carried Computing unit data packet；Center-side setup module, for being arranged in corresponding computing unit according to the computing unit data packet Heart end, and port is distributed for the computing unit center-side of setting；Sending module, for by the computing unit data packet and The port numbers for distributing to the port of the computing unit center-side are sent to the multiple node server；
The node server includes: node side setup module, after receiving the computing unit data packet, setting Corresponding computing unit node side；Connection establishment module, by be by the corresponding port of the port numbers setting it is described based on It calculates cell node end and the computing unit center-side establishes connection.
With reference to first aspect, the embodiment of the invention provides the 4th kind of possible embodiment of first aspect, it is described in Central server includes: deletion instruction acquisition module, deletes instruction for obtaining computing unit, the computing unit is deleted in instruction Carry the mark of computing unit；Computing unit center-side Unload module is taken for being deleted in instruction according to the computing unit The mark of the computing unit of band unloads the corresponding computing unit center-side of mark of computing unit；It deletes instruction and issues module, use Instruction is deleted in sending the computing unit to the multiple node server；
The node server includes: computing unit node side Unload module, is deleted for that ought receive the computing unit When except instruction, according to the mark of the computing unit in the computing unit delete command, the mark pair of the computing unit is unloaded The computing unit node side answered.
With reference to first aspect, the embodiment of the invention provides the 5th kind of possible embodiment of first aspect, it is described in Central server includes: common tasks interface, and by the common tasks interface, obtain user's transmission has different task type The video image task.
Second aspect, the embodiment of the present invention provide a kind of realizing using above-mentioned distributed computing resource distribution system for task Processing method, comprising:
The central server receives the video image task that user sends, according to the type of the video image task, And video image task quantity and loading condition that the multiple node server is current, the video image task is passed through Computing unit center-side distributes to corresponding node server；
After the node server receives the video image task, pass through the corresponding calculating of the video image task Cell node end handles the video image task.
In conjunction with second aspect, the embodiment of the invention provides the first possible embodiments of second aspect, wherein institute Central server is stated to be appointed according to the current video image of the type of the video image task and the multiple node server Quantity of being engaged in and loading condition, distribute to corresponding node server packet by computing unit center-side for the video image task It includes:
The central server determines according to the task quantity of distribution of node server each in current server cluster The least node server of distribution task quantity；
The central server selects the least node server of task quantity corresponding as the video image task Node server；
When the task quantity of distribution for having multiple node servers is minimum, the central server selection is the multiple The minimum node server of resource utilization is as the corresponding node server of the video image task, institute in node server State the utilization rate that resource utilization comprises at least one of the following server hardware resource: central processing unit, memory and network bandwidth； Wherein, the node server periodically feeds back current resource utilization to the central server.
In conjunction with second aspect, the embodiment of the invention provides second of possible embodiments of second aspect, wherein institute State method further include:
The central server searches the task list of the corresponding node server of the video image task, at described The video image task is added in the task quantity of distribution of list of being engaged in；
When receiving the completed information of the video image task that the computing unit node side is sent, it is described in Video image task described in the task quantity of distribution of the central server by the task list is deleted；
When the video image task is completed, the node server passes through the corresponding calculating of the video image task Cell node end sends the completed information of task to the central server, and the completed information of task carries correspondence Computing unit where node server mark.
In conjunction with second aspect, the embodiment of the invention provides the third possible embodiments of second aspect, wherein institute State method further include:
The central server obtains computing unit addition instruction, and the computing unit addition instruction carries computing unit Data packet；
Corresponding computing unit center-side is arranged according to the computing unit data packet in the central server, and to set The computing unit center-side distribution port set；
The central server is by the computing unit data packet and the port for distributing to the computing unit center-side Port numbers are sent to the multiple node server；
After the node server receives the computing unit data packet, corresponding computing unit node side is set；
The node server is the computing unit node side of setting and institute by the corresponding port of the port numbers It states computing unit center-side and establishes connection.
In conjunction with second aspect, the embodiment of the invention provides the 4th kind of possible embodiments of second aspect, wherein institute State method further include:
The central server obtains computing unit and deletes instruction, and it is single to carry calculating in the computing unit deletion instruction The mark of member；
The central server deletes the mark of computing unit carried in instruction according to the computing unit, described in unloading The corresponding computing unit center-side of the mark of computing unit；
The central server sends the computing unit to the multiple node server and deletes instruction；
When receiving the computing unit deletion instruction, the node server is according to the computing unit delete command In computing unit mark, unload the corresponding computing unit node side of mark of the computing unit.
In conjunction with second aspect, the embodiment of the invention provides the 5th kind of possible embodiments of second aspect, wherein institute Central server is stated by the common tasks interface, obtains the video image with different task type that user sends Task.
A kind of distributed computing resource distribution system provided in an embodiment of the present invention and task processing method, pass through setting Central server receives the video image task that user sends, according to the current task of node servers multiple in server cluster The computing unit center-side that video image task is arranged by central server is distributed to server set by quantity and loading condition Pass through video image task in computing unit node side corresponding with the computing unit center-side in any node server in group Corresponding computing unit node side handles video image task；Video image is appointed by being arranged in central server The computing unit center-side that business is allocated, and setting can handle respective type in each server of server cluster The computing unit node side of video image task, so as to use multiple servers an of server cluster that can handle Different types of video image task, the process demand for being just able to satisfy video image task without disposing multiple server clusters, The utilization rate for improving server cluster reduces the construction cost of server cluster, and passes through a server cluster Multiple different types of video image tasks of server process, improve the utilization rate of server resource in server cluster, keep away The waste of resource is exempted from.
To enable the above objects, features and advantages of the present invention to be clearer and more comprehensible, preferred embodiment is cited below particularly, and cooperate Appended attached drawing, is described in detail below.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this A little attached drawings obtain other relevant attached drawings.
Fig. 1 shows a kind of structural schematic diagram of distributed computing resource distribution system of the offer of the embodiment of the present invention 1；
Cell processing is calculated in a kind of distributed computing resource distribution system provided Fig. 2 shows the embodiment of the present invention 1 Task type schematic diagram；
Fig. 3 shows the framework signal of another kind distributed computing resource distribution system provided by the embodiment of the present invention 2 Figure；
Fig. 4 show the embodiment of the present invention 2 offer distributed computing resource distribution system in video image task into The computing unit node side schematic diagram that row historical data calculates；
Fig. 5 show the embodiment of the present invention 2 offer distributed computing resource distribution system in video image task into The computing unit node side schematic diagram calculated in row memory；
Fig. 6 show the embodiment of the present invention 2 offer distributed computing resource distribution system in video image task into The computing unit node side schematic diagram of row real time data parallel computation；
Fig. 7 show the embodiment of the present invention 2 offer distributed computing resource distribution system in video image task into The computing unit node side schematic diagram that row high concurrent small block data calculates；
Fig. 8 shows the flow chart of the task processing method of the offer of the embodiment of the present invention 3.
Specific embodiment
Below in conjunction with attached drawing in the embodiment of the present invention, technical solution in the embodiment of the present invention carries out clear, complete Ground description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.Usually exist The component of the embodiment of the present invention described and illustrated in attached drawing can be arranged and be designed with a variety of different configurations herein.Cause This, is not intended to limit claimed invention to the detailed description of the embodiment of the present invention provided in the accompanying drawings below Range, but it is merely representative of selected embodiment of the invention.Based on the embodiment of the present invention, those skilled in the art are not doing Every other embodiment obtained under the premise of creative work out, shall fall within the protection scope of the present invention.
In view of in the related technology, it is existing video is handled as unit of server cluster in the way of in, one Server cluster can only realize a kind of video processing mode, when the processing mode to video has very much, must just dispose quantity Many server clusters just can satisfy the process demand of video, thus just considerably increase the construction of server cluster at This, moreover, the utilization rate of each server cluster is different, and the server resource in the low server cluster of utilization rate cannot It is used by other video processing modes, causes the waste of resource.Based on this, the embodiment of the invention provides a kind of distributions Formula computational resource allocation system and distributed approach.
Embodiment 1
Referring to Fig. 1, the present embodiment provides a kind of distributed computing resource distribution systems, comprising: central server 100 and more A node server 110, wherein the computing unit center-side for different task type is provided in central server 100 101, each node server 110 is respectively arranged with computing unit node side 111 corresponding with computing unit center-side 101, meter It calculates unit center end and corresponding computing unit node side forms a computing unit；
Central server 100 is used to receive the video image task of user's transmission, according to the type of video image task, with And task quantity and loading condition that multiple node servers 110 are current, video image task is passed through into computing unit center-side 101 distribute to corresponding node server 110；
Node server 110 is used to appoint video image by the corresponding computing unit node side 111 of video image task Business is handled.
The schematic diagram of the task type handled from computing unit is as shown in Fig. 2, each computing unit can be to specific type Video image task handled, so central server 100 receive user transmission video image task when, all can The task type carried in the video image task received is first obtained, according to the task type of video image task, will be received To video image task distribute to processing corresponding task type computing unit computing unit center-side 101.
Central server 100 and node server 110, which can be used, existing any can carry out distributed computing resource Distribution and the server and terminal for carrying out distributed computing processing, no longer repeat one by one here.
Video image task, including video task and image task, wherein video task refer to user issue to video Or the task that video image is handled；Image task refers to that user issued handles video or video image Task；Since video task and image task are similar in processing, so being arranged in distributed computing resource distribution system Computing unit video task and image task can be carried out while being handled.
Video image task includes but is not limited to: vehicle comparison, face alignment, video code conversion, image rotation, image defogging It deploys troops on garrison duty with video.Wherein, image rotation and image defogging two processing can be to the video image in video task, can also be with It is the processing carried out to the image in image task.
The content of video handled by video image task and image is got by camera or video camera, Here it no longer repeats one by one.
Video image task can be divided into different task types according to the difference of the processing mode to video image. When identical there are many processing mode of video image task, which can be divided into a kind of video figure As task.
Division to the task type that video image task carries out, can be when developing distributed computing resource distribution system Preset, can also by distributed computing resource distribution system user using one immediately between after, find to some It, can be by the option of distributed computing resource distribution system to these when the task type of video image task divides inaccuracy The task type of video image task is repartitioned.
Such as due to in video image face comparison and vehicle compare when, require the face in the image that will acquire Or vehicle is compared with storage face in the database and vehicle, then the speed in order to guarantee comparison, in database Face and model data should retain for a long time in memory, can be with real-time perfoming people with when needing face and vehicle to compare The analysis of face and vehicle, thus quickly to user feedback comparison as a result, the face pair in video image so can be set Than being identical task type with the task type that vehicle compares, carried out by the computing unit calculated in memory can be carried out Reason.
Task type includes but is not limited to: it is parallel that type, historical data parallel computation type, real time data are calculated in memory It calculates type and high concurrent small block data calculates type.
For different types of video image task, it is respectively processed by different computing units.When distribution is counted Calculating the manageable task type of resource allocation system includes that type, historical data parallel computation type, in real time are calculated in memory When data parallel type and high concurrent small block data calculate type, will include: in distributed computing resource distribution system For the first computing unit of calculation processing in memory, for the second computing unit of historical data parallel computation, for real-time The third computing unit of data parallel and the 4th computing unit calculated for high concurrent small block data.
Each computing unit includes the computing unit center-side 101 for being used to distribute video image task and one Computing unit node side 111 for being handled video image task；Computing unit node side in each computing unit 111 are connected by scheduled port with corresponding computing unit center-side 101, to receive the transmission of computing unit center-side 101 Video image task, and when video image task is completed, task completed information is fed back to computing unit center-side 101.
Distributed computing resource distribution system provided in this embodiment receives user by the central server of setting and sends Video image task, according to the current task quantity of node servers multiple in server cluster and loading condition, by video The computing unit center-side that image task is arranged by central server is distributed in server cluster in any node server In computing unit node side corresponding with the computing unit center-side, pass through the corresponding computing unit node side of video image task Video image task is handled；By the way that the computing unit being allocated to video image task is arranged in central server Center-side, and setting can handle the calculating list of respective type video image task in each server of server cluster First node side is appointed so as to use multiple servers an of server cluster that can handle different types of video image Business, the process demand for being just able to satisfy video image task without disposing multiple server clusters improve the benefit of server cluster With rate, the construction cost of server cluster is reduced, and passes through multiple server process inhomogeneities of a server cluster The video image task of type, improves the utilization rate of server resource in server cluster, avoids the waste of resource.
In the related technology, the server that is allocated to task can the task type of video image task based on the received Difference receives different types of task that user sends over by different interfaces, so in the early development of server When, need to develop different interfaces for different task types, in order to reduce exploitation interface quantity, center service Device is preset with the common tasks interface of the video image task with different task type for obtaining family transmission, and user sends Video image task can be obtained by common tasks interface by central server.
By above description as can be seen that the common tasks interface by being arranged, what reception user sent has difference The task of task type improves the development rate of software without separately designing interface to each computing unit.
User to central server send video image task when, user by using terminal to central server send out Video image task to be treated is sent, the task type of video image task, center service are carried in video image task Device, can appointing according to the video image task carried in video image task when getting the video image task of user's transmission Service type in central server in the corresponding relationship of the mark of preset task type and computing unit, determines and handles this Then the video image task is distributed to the video figure for handling the task type by the computing unit of service type video image task As the computing unit center-side of task；And the computing unit center-side will send server resource application to central server and ask It asks, central server, will be according to multiple after getting the server resource application request of computing unit center-side transmission Video image task is distributed to correspondence by computing unit center-side by the current task quantity of node server and loading condition Node server.
In the related technology, the server being allocated to task can only assign the task to clothes in the task of distribution at random It is engaged in any server in device cluster, will lead to the load imbalance of each server in server cluster, to reduce service The task processing capacity of device cluster, central server pass through setting task quantity determining module, first choice module and the second choosing It is uneven come the load for alleviating each server in server cluster to select the information reporting module being arranged in module and node server Weigh situation, and central server specifically includes: task quantity determining module, for according to node serve each in current server cluster The task quantity of distribution of device, determination have distributed task quantity least node server；First choice module is appointed for selecting The node server for minimum number of being engaged in is as the corresponding node server of video image task；Second selecting module has for working as When the task quantity of distribution of multiple node servers is minimum, the section that resource utilization is minimum in multiple node servers is selected Point server comprises at least one of the following server hardware as the corresponding node server of video image task, resource utilization The utilization rate of resource: central processing unit, memory and network bandwidth.
Node server specifically includes: information reporting module, and current resource is fed back to central server for periodicity Utilization rate.
Resource utilization data of the central server in the current each server for receiving node server periodic feedback It afterwards, can be by the data buffer storage of the resource utilization of the current each server received.Since node server is periodically Current resource utilization is fed back to central server, then central server can connect again when preset duration reaches The data of the resource utilization of current each server of node server feedback are received, central server can be by the money of caching at this time The data of source utilization rate wipe out, then will be in the data write-in caching of the resource utilization for each server that be currently received.
Task quantity determining module, specifically includes: task number obtainment unit, for obtaining each node in server cluster The task quantity of distribution of server；Task quantity comparing unit, for comparing the task quantity of distribution of each node server, Determine the minimum value for having distributed task quantity；Node server determination unit, for selecting to have distributed the minimum value of task quantity Corresponding node server, which is used as, has distributed task quantity least node server.
When the task quantity of distribution for currently having multiple node servers when task quantity determining module is determining is minimum, the Two selecting modules, specifically include: resource utilization acquiring unit, for obtaining the distribution number of tasks cached in central server Measure the data of the resource utilization of least multiple node servers；Resource utilization comparison unit has been distributed for comparing The height of the resource utilization of the least multiple node servers of task quantity, determines the minimum of resource utilization；Node Server determination unit, for select the corresponding node server of minimum of resource utilization as the task quantity of distribution most Few node server.
Server mainly passes through central processing unit and handles video image task, then resource utilization comparison unit exists When comparing the height for the resource utilization for having distributed the least multiple node servers of task quantity, preferably by comparing each clothes The utilization rate of each core of central processing unit in business device determines the minimum node server of resource utilization.
Task quantity comparing unit and resource utilization comparison unit can be using existing any relatively population sizes Method determines the minimum of the minimum value and resource utilization of having distributed task quantity respectively, no longer repeats one by one here.
It can be seen that by above description when distributing task to each server, by the principle of load balancing to appointing Business is allocated, can be according to the service condition of server each in server cluster, in the task of distribution, in server cluster Each server carry out unified management and distribution.
For the task according to the loading condition of server to each server allocation processing, central server is appointed by setting Business adding module and the task feedback module of task removing module and node server setting grasp each server in real time Loading condition.Central server specifically includes: task adding module, for searching the corresponding node server of video image task Task list, in the task quantity of distribution of task list add video image task；Task removing module connects for working as When receiving the completed information of video image task of computing unit node side transmission, by the task quantity of distribution of task list Middle video image task is deleted；
Node server includes: task feedback module, for passing through video image task when video image task is completed Corresponding computing unit node side sends the completed information of task to central server, and the completed information of task carries pair The mark of node server where the computing unit answered.
Record has the corresponding node of mark of the mark and node server of each node server in each task list Server has distributed task quantity, handles video by having distributed the least node server of task quantity when central server is determined When image task, central server will add video in the task list for having distributed the least node server of task quantity Image task, to be updated to the task list for having distributed the least node server of task quantity.
When the completed information of the video image task for receiving the transmission of computing unit node side, task removing module root According to the mark of the node server recorded in the completed information of video image task, the mark for inquiring node server is corresponding Node server task list, task is arranged in the task list of the corresponding node server of mark of node server Video image task is deleted in the task quantity of distribution of table, to be updated to task list.
After receiving the completed information of video image task, central server can be by common tasks interface to user The message that video image task is completed is sent, and notifies that user is disposed the storage of rear video image task processing result Location allows user to obtain the processing result of video image task by storage address.
By above description as can be seen that recording the number of tasks that each server is distributed by preset task list Amount, so that it may the loading condition of each server is determined in real time, to distribute according to the loading condition of server to each server The task of processing can make the load of server more balanced.
As video handles the development with video image processing technology, user is more next to the process demand of video image task More, the computing unit of existing setting cannot be to certain task types that user issues in distributed computing resource distribution system When video image task is effectively treated, it is necessary to new computing unit be set in central server and come to these video figures As task is handled, so by the way that instruction acquisition module, center-side setup module and sending module is arranged in central server, And it is set in central server and node server in node server setting node side setup module and connection establishment module Set new computing unit, central server specifically includes: instruction acquisition module is calculated for obtaining computing unit addition instruction Unit addition instruction carries computing unit data packet；Center-side setup module, for according to computing unit data packet, setting pair The computing unit center-side answered, and port is distributed for the computing unit center-side of setting；Sending module is used for computing unit number Multiple node servers are sent to according to the port numbers for the port for wrapping and distributing to computing unit center-side.
Node server specifically includes: node side setup module, and after receiving computing unit data packet, setting is corresponded to Computing unit node side；Connection establishment module, for being the computing unit node side of setting by the corresponding port of port numbers Connection is established with computing unit center-side.
When the maintenance personnel of distributed computing resource distribution system has found certain type of video image task by distribution The spent processing time is longer when existing all computing units are handled in computational resource allocation system, cannot to user and When feedback processing result when, this will can be effectively treated according to the processing feature of such video image task, determination Then the computing unit of the video image task of seed type obtains the computing unit of the video image task of processing this type Computing unit data packet, and the computing unit data packet by getting forms computing unit addition instruction, by the calculating of formation Unit addition instruction is sent to central server, so that corresponding calculating is arranged according to computing unit data packet in central server Unit center end.
The maintenance personnel of distributed computing resource distribution system can obtain from system background server handles the type The computing unit data packet of the computing unit of the video image task of type can also pass through the view of network download process this type The computing unit data packet of the computing unit of frequency image task, can also be by existing any data packet acquisition modes, to obtain Computing unit data packet is taken, is no longer repeated one by one here.
Certainly, the computing unit center-side for each computing unit being arranged in central server can also be to handled video The processing time of image task is recorded, and when task processing is completed, by the processing time of video image task and task Completed information feeds back to central server together, and central server can be by processing time of video image task and default Processing time threshold compare, when the processing time of the video image task be greater than preset processing time threshold when, say Bright current computing unit is not suitable for handling such video image task, then central server can be by the view The task type of frequency image task is associated with other computing units, so that appoint to such video image next time When business is handled, it can be handled with other computing units, when central server determines that the computing unit of current setting is equal When being not suitable for handling such video image task, it will be determined from system background server and obtain processing The computing unit data packet of the computing unit of the video image task of this type, and the computing unit data packet by getting Computing unit addition instruction is formed, the computing unit addition instruction of formation is sent to central server, so that central server According to computing unit data packet, corresponding computing unit center-side is set.
It is single according to computing unit addition instruction setting calculating is obtained to can be seen that central server by above description Member, and the process for adding computing unit is simple, convenient and fast, it is flexible and convenient to use, and after adding new computing unit, The processing time of video image task can be reduced and task can be handled in time.
As video handles the development with video image processing technology, existing setting in distributed computing resource distribution system Some computing units handled by video image task it is fewer and fewer or even some computing units have been no longer used, still These computing units being no longer used still can occupy central server and the resource of node server causes server resource Waste, so instruction acquisition module, computing unit center-side Unload module and being deleted by being arranged to delete in central server The meter not used is unloaded except instruction issues the computing unit node side Unload module that is arranged in module and node server Calculate unit.Central server specifically includes: deleting instruction acquisition module, deletes instruction, computing unit for obtaining computing unit Delete the mark that computing unit is carried in instruction；Computing unit center-side Unload module, for being referred to according to computing unit deletion The mark of the computing unit carried in order unloads the corresponding computing unit center-side of mark of computing unit；Instruction is deleted to issue Module deletes instruction for sending computing unit to multiple node servers.
Node server includes: computing unit node side Unload module, for when receive computing unit delete instruction when, According to the mark of the computing unit in computing unit delete command, the corresponding computing unit node of mark of computing unit is unloaded End.
When the maintenance personnel of distributed computing resource distribution system has found certain type of video image task for a long time not When reprocessing task, the computing unit that the mark for carrying computing unit will be sent to central server deletes instruction, so that Central server and node server delete the corresponding computing unit of mark of computing unit.
Certainly, when central server can also distribute the last time to the task of computing unit distribution video image task Between recorded, and can periodically judge the last task distribution time to computing unit distribution video image task with Whether the time interval of current time has reached the erasing time of computing unit, if having reached the erasing time of computing unit, Instruction just so is deleted to the computing unit that central server sends the mark for carrying the computing unit that can be deleted, so that in Central server and node server delete the corresponding computing unit of mark of computing unit.
The maintenance personnel of distributed computing resource distribution system can be by the input equipment of central server or in The input equipment of the system background server of central server connection sends computing unit addition instruction to central server and calculates Element deletion instruction.
By above description as can be seen that for not in the computing unit used, the needs that can be handled according to task It is deleted, it is flexible and convenient to use, and also unloading computing unit is convenient and efficient, so that in central server and node server Resource can will not influence other processing units to the treatment process of task by reasonable utilization.
Embodiment 2
Referring to Fig. 3, another distributed computing resource distribution system, the table in the form of CUMN framework are present embodiments provided Reveal and, distributed computing resource distribution system includes center (Center), user (User), module (Module), model (Model) and 5 parts of node (Node).Different business demands also can be different for the processing mode of data, and this system is adopted Different calculations is defined with model, there is the meaning of modeling.For example, Hadoop use is exactly that Map/Reduce calculates mould Type.CUMN framework provides that every kind of computation model is made of one or several centers and several nodes.Model in frame is " heat Plug ", it can arbitrarily add under operation or case-deleted models, this greatly improves distributed computing resource distribution The scalability and compatibility of system.
Model is a kind of calculation, specifically calculates that this programme is defined as module, it is envisaged that at one kind Algorithm.Video structural, video concentration, video frequency searching etc. are realized for example, by using grid, each function is all a module.Net It is only defined in lattice model and splits operation by the way of Map/Reduce, but how calculated after splitting, be the function of module Energy.In the present solution, module is also " hot plug " design, it can dynamically add and removing module, be mentioned for System Expansion, upgrading Convenience is supplied.
All models in this programme can independently become product, such as grid.It can be adopted in the system of access With the publicly-owned interface of frame, the privately owned interface that can also be provided using model itself.This programme determines the part interacted with user Justice is U, and the U of capitalization represents the public interface of frame, the privately owned interface of u representative model of small letter.The publicly-owned interface of frame sometimes can not When meet demand, the privately owned interface of model can be used.
CUMN framework makes distributed computing resource distribution system become general frame, and actual calculating is then by mould What type and module were realized.The characteristics of video, image analysis, is as follows:
Data volume is big: the data that each city such as video monitoring, traffic block port generates tens T to several P daily differ, public security Settling a case is the video needs quickly analysis that can also draw a circle to approve tens road cameras；
Data type and processing mode are more: history video, image analysis, real-time video, image analysis.
Model is a kind of processing model of calculating, such as the MapReduce of Hadoop of mainstream is exactly a kind of big number of processing According to computation model.But in field of video applications, since the diversity (video, image, target signature) of data type makes certain A kind of computation model can not adapt to total data type, such as Hadoop is suitble to handle (GB~TB grades of a biggish file Not), one is only had the picture of several hundred KB ranks with regard to helpless.Response speed of the business different simultaneously for data processing It is required that also different (such as image procossing, face alignment will be returned quickly, and video frequency searching can slowly go out result), more in order to cope with Kind data and type of service, this programme propose the concept of model.In the present solution, model is to handle a certain data or business Calculation method, all computation models both define center-side (with small letter c) and node side (with small letter c), the center of model End is managed by the center (with capitalization C) of Computational frame, and distributes unique port numbers for connecting with the node side of model for it； The node side of model has node side (with the N of capitalization) management of frame.
Hypothesized model 1 will give each node to calculate after each task cutting；Model 2 is that each node calculates a task； Model 3 is that all calculating have to load data into memory；It is thus relatively good that the concept for having model why understood.
This programme uses model hot plug mode, can according to need online addition model, the center of newly added model End to central node server and is started by automatic deployment, and the starting of model center end is to be assigned a port number, model center End program monitors the port numbers, the node side connection of Holding Model；The node section program of model can be by automatic deployment to frame tube It has jurisdiction in whole node servers of range, when Boot Model node side program, may be notified that the port numbers that model center end is monitored, This completes the automatic additions and deployment of model.
N:N mode when program central server and node server, i.e., a large amount of node server with no less than one Central server, central server be greater than 1 can be to avoid Single Point of Faliure problem；Each central server needs to dispose we The centre management software of pattern frame, each node server need to dispose the node administration software of our pattern frame；Centre management Software is responsible for managing the center-side program (holding program to distribute unique port centered on including) of all models, node administration software It is responsible for the node side program of all models of management.
How the data that model defines utilize cluster to work, but specifically What for this programme are wanted to propose the concept of module. For the utilization rate of lift scheme, calculation process is all module that is identical, only loading anyway in this way for model Different (such as image enhancement and image rotations).There is the concept of module, model be not concerned about oneself what does, Different resume modules is only called as needed, specifically how to be handled, is indifferent to.In this way, according to new business needs are increased Increase a module, this is also the reason of this programme is referred to as frame.
C and N in model management not only realize the function of automatic deployment, while the money of each model of collection of meeting active Source use state, to realize the unified load balancing of resource in cluster.Each model will be referring to when dispatching resource Unified load in cluster will be run in mission dispatching to the relatively low node of load.
A variety of computation models are deployed in same set of cluster by traditional approach, and due to not communicating between model, model is certainly Think that node A is more idle, it is full to hardly realize that the resource of node A has been used by other models.
Distributed computing resource distribution system externally provides a set of using interface, and user is allowed to use all moulds in frame Block.In order to promote the friendly of this programme, user need not know the concept of model and module, by interface it can be seen that in frame Have those of which company algorithm, and the request of oneself is realized using the algorithm.This requires frames being capable of automatic identification The affiliated model and trunk of algorithm, as already explained above, details are not described herein again for this point.
This point is also compared to the more friendly of a variety of computation model clusters is simply stacked, because of a variety of computation model clusters Interface mode, language all may be different, it is not only inconvenient to use, while increasing new model every time and requiring outer exploitation work Make, and this programme then only needs to develop once, it is also constant for increasing model interface.
Distributed computing resource distribution system can carry out following calculation processing to video image task:
Fig. 4 is the computing unit node side schematic diagram that historical data calculating is carried out to video image task, historical data meter Integrated computation model is defaulted in this frame at last, how it just can solve a very huge computing capability of needs if being studied The problem of be divided into many small parts, many computers are then distributed in these parts and are handled, finally these are calculated As a result it integrates to obtain final result, Grid Computing Model in corresponding this programme.There are one Grid Computing Models, and characteristic is Idle computing capability is made full use of, even if can generally also effectively improve physics under the premise of not adding new computing capability Machine hardware utilization.
Fig. 5 is the computing unit node side schematic diagram for calculate in memory to video image task, is calculated in memory (In-Memory Computing) is the integrated computation model of this frame default, is substantially exactly CPU hard directly from memory Data are read on disk, and data are calculated, are analyzed.Technique is a kind of acceleration to traditional data processing mode, is It realizes intelligent mass data analysis and implements the crucial application technology of data analysis.
The data for being very suitable to processing magnanimity are calculated in memory, and need to obtain the data of result in real time.Such as it can be with The data of the various aspects such as the face characteristic in resident population library, stolen vehicle information are disposably stored in memory, and herein On the basis of carry out data analysis.When needing to do quick data analysis, memory calculating can be completed quickly as desired.
In terms of calculating real-time, in order to reduce data-moving bring expense, memory uses the side of data memory-resident Method.Meanwhile each calculate node only calculates the data of this node memory, the 0 data load in basic guarantee system response cycle Process.In order to promote calculating handling capacity, each node only calculates the data of this node memory storage as far as possible, drops to greatest extent Data-moving expense between low node.The data of this programme use the storage strategy of redundancy, and in node exception, other nodes can Automatically to load the availability that copy guarantees system.Since the data volume for being suitble to memory to calculate is first to less big, generally several Ten G to several hundred G or so will not bring pressure to storage system by the way of copy storage.
Fig. 6 is the computing unit node side schematic diagram that real time data parallel computation is carried out to video image task, in real time number It is the integrated computation model of this frame default according to parallel computation, analyzes video (picture) stream, such as video deployment in real time.View in real time Frequently the timely extraction for solving various events in live video stream is absorbed in (picture) analysis.
Fig. 7 is the computing unit node side schematic diagram that the calculating of high concurrent small block data is carried out to video image task, and height is simultaneously It is the integrated computation model of this frame default, system different from the data of parallel computation processing and mode that hair small block data, which calculates, In there is also the small block datas of magnanimity to need to handle.The timeliness of these data processings is more demanding, exists simultaneously height simultaneously The possibility of hair, typical application is image procossing.This programme is the place of such scheduling, equilibrium for calculating and devising consistency Hash Reason mode, not only can guarantee the timely processing of data, but also can solve high concurrent request problem.
The distributed computing resource distribution system that the present embodiment proposes, can supervise the physical resource in calculate node Control and dynamically distribute, each calculate node can installation frame node background program, which can obtain the category of calculate node Property information, including CPU frequency, CPU core number, CPU line number of passes, memory amount, OS Type, network bandwidth etc., and report To the center of frame.Meanwhile node procedure can also periodically (be defaulted as 1 second, can configure) reporting node cpu, memory, net to center The utilization rate of network, centered on allotment resource provide reference data.
The request that user submits every time is defined as operation in the frame, and operation is split into multiple fragments in multiple physical machines Middle execution, each fragment frame are defined as task.The scheduling minimum unit of frame is task, cutting of the frame for physical resource For CPU core (including hyperthread), each task is operated on a CPU core with independent process (although operating system can will be into Journey switches in multicore, but from the service condition of resource is the computing capability of a core)
Distributed computing resource distribution system, may operate in physical machine and virtual machine, may operate in simultaneously In Windows, Linux, Unix operating system.Physical machine range from mainframe, minicomputer, server, individual PC, notebook, Tablet computer, smart phone etc. calculate equipment, while can also accelerate operation using video card, the coprocessor integrated in physical machine.
The connection relationship of various pieces in distributed computing resource distribution system and the function of being realized are made into one below The description of step.
Distributed computing resource distribution system includes: center (hereinafter referred to as central server), calculate node (hereinafter referred to as node Server), computing unit center-side and computing unit node side.
Central server receives the waiting task that user sends for the task interface by setting, and according to wait locate Waiting task is sent to the computing unit center-side of processing corresponding task by the task type of reason task；It is calculated receiving When the resource allocation request that unit center end is sent, according to the processing task of each node server in current distributed computing cluster Quantity and loading condition, determine the node server of distribution, and the node server distributed is fed back to computing unit center End, and according to this resource allocation as a result, updating preset task assignment list；It is fed back when receiving computing unit center-side Task when discharging request, according to the mark of node server carried in task release request, update task list, and pass through Task interface sends task to user and information is completed；When the needs handled according to task, when needing to add new computing unit, Central server can get computing unit addition instruction, according to the computing unit data carried in computing unit addition instruction Packet installs newly added computing unit center-side, and to newly added computing unit center-side distribution computing unit mark, then Each node server is sent by computing unit mark and data packet, allows each node server that the newly added meter is installed Cell node end is calculated, and distributes port to the newly added computing unit center-side, to establish computing unit center-side and calculating The connection at cell node end.When certain computing unit does not use and needs to delete the computing unit, central server can be obtained It gets computing unit and deletes instruction, the computing unit carried in instruction is deleted according to computing unit and is identified, computing unit mark is deleted Know corresponding computing unit center-side, is then sent to each node server and carry the computing unit deletion that computing unit identifies Order, so that each node server deletes computing unit and identifies corresponding computing unit.
Computing unit center-side, in centrally disposed server, for receiving the waiting task of central server transmission, After receiving waiting task, resource allocation request is sent to central server, receives the resource point that central server is sent With information, and by task to be processed by preset port be sent in resource allocation information specify node server in The corresponding computing unit node side of the computing unit center-side carries out the processing of task；It is sent when receiving computing unit node side Task complete request when, to central server send carry node server mark task discharge request, with update The task assignment list prestored in central server；When newly addition computing unit, newly added computing unit center-side can lead to Instruction is established in the connection for the computing unit node side that the port for crossing central server distribution monitors each node server installation, is being connect It receives after calculating the connection foundation instruction that cell node end is sent in each node server, establishes and calculated in each node server The connection at cell node end.
Node server works as prosthomere to central server feedback for handling the task of distribution, and periodically Loading condition in point server allows central server according to the loading condition of each node server, takes to each node Business device distributes task；When newly addition computing unit, the data packet that can be sent according to central server installs newly added calculating The computing unit node side of unit, and the computing unit mark sent to the computing unit node side distributing center server of installation Know, and informs that the node side of newly added computing unit is established with the computing unit center-side with identical calculations unit marks Port when connection, so that newly added computing unit node side passes through specified port；When deleting computing unit, node clothes The computing unit mark carried in the computing unit delete command that business device is sent according to central server, it is single to delete corresponding calculating First node side.
Computing unit node side is arranged in each node server, for handling the task of distribution, at task When reason finishes, task completed information is sent to corresponding computing unit center-side；In the computing unit for adding new computing unit When node side, the port and computing unit center-side that newly added computing unit node side is specified by node server, which are established, to be connected It connects.
Distributed computing resource distribution system provided in this embodiment receives user by the central server of setting and sends Video image task, according to the current task quantity of node servers multiple in server cluster and loading condition, by video The computing unit center-side that image task is arranged by central server is distributed in server cluster in any node server In computing unit node side corresponding with the computing unit center-side, pass through the corresponding computing unit node side of video image task Video image task is handled；By the way that the computing unit being allocated to video image task is arranged in central server Center-side, and setting can handle the calculating list of respective type video image task in each server of server cluster First node side is appointed so as to use multiple servers an of server cluster that can handle different types of video image Business, the process demand for being just able to satisfy video image task without disposing multiple server clusters improve the benefit of server cluster With rate, the construction cost of server cluster is reduced, and passes through multiple server process inhomogeneities of a server cluster The video image task of type, improves the utilization rate of server resource in server cluster, avoids the waste of resource.
Embodiment 3
Referring to Fig. 8, the present embodiment provides a kind of task processing methods, can be provided by the distributed computing in above-described embodiment Source distribution system implement, the task processing method the following steps are included:
Step 800, central server receive the video image task that user sends, according to the type of video image task, And video image task quantity and loading condition that multiple node servers are current, video image task is passed through into computing unit Center-side distributes to corresponding node server；
After step 802, node server receive video image task, pass through the corresponding computing unit of video image task Node side handles video image task.
In the related technology, the server that is allocated to task can the task type of video image task based on the received Difference receives different types of task that user sends over by different interfaces, so in the early development of server When, need to develop different interfaces for different task types, in order to reduce exploitation interface quantity, center service Device is preset with the common tasks interface of the video image task with different task type for obtaining family transmission, and user sends Video image task can be obtained by common tasks interface by central server.
By above description as can be seen that the common tasks interface by being arranged, what reception user sent has difference The task of task type improves the development rate of software without separately designing interface to each computing unit.
In the related technology, the server being allocated to task can only assign the task to clothes in the task of distribution at random It is engaged in any server in device cluster, will lead to the load imbalance of each server in server cluster, to reduce service The task processing capacity of device cluster, in order to guarantee the load balancing of each node server, central server is appointed according to video image The type of business and the current video image task quantity and loading condition of multiple node servers, video image task is led to It crosses computing unit center-side and distributes to corresponding node server step and include the following steps (1) to step (3):
(1) central server determines according to the task quantity of distribution of node server each in current server cluster The least node server of distribution task quantity；
(2) central server selects the least node server of task quantity as the corresponding node clothes of video image task Business device；
(3) when the task quantity of distribution for having multiple node servers is minimum, central server selects multiple nodes The minimum node server of resource utilization is as the corresponding node server of video image task, resource utilization in server Comprise at least one of the following the utilization rate of server hardware resource: central processing unit, memory and network bandwidth；Wherein, node takes Business device periodically feeds back current resource utilization to central server.
It can be seen that by above description when distributing task to each server, by the principle of load balancing to appointing Business is allocated, can be according to the service condition of server each in server cluster, in the task of distribution, in server cluster Each server carry out unified management and distribution.
For the task according to the loading condition of server to each server allocation processing, central server is according to video figure As the type of task and the current video image task quantity and loading condition of multiple node servers, video image is appointed It further includes including the following steps (4) to step (6) that business, which distributes to corresponding node server step by computing unit center-side:
(4) central server searches the task list of the corresponding node server of video image task, in task list Addition video image task in task quantity is distributed；
(5) when the completed information of the video image task for receiving the transmission of computing unit node side, central server Video image task in the task quantity of distribution of task list is deleted；
(6) node server passes through the corresponding computing unit node of video image task when video image task is completed It holds to central server and sends the completed information of task, the completed information of task carries section where corresponding computing unit The mark of point server.
By above description as can be seen that recording the number of tasks that each server is distributed by preset task list Amount, so that it may the loading condition of each server is determined in real time, to distribute according to the loading condition of server to each server The task of processing can make the load of server more balanced.
As video handles the development with video image processing technology, user is more next to the process demand of video image task More, the computing unit of existing setting cannot be to certain task types that user issues in distributed computing resource distribution system When video image task is effectively treated, it is necessary to new computing unit be set in central server and come to these video figures As task is handled, so in order to add computing unit, task processing method further comprising the steps of (1) to step (5):
(1) central server obtains computing unit addition instruction, and computing unit addition instruction carries computing unit data Packet；
(2) corresponding computing unit center-side is arranged according to computing unit data packet in central server, and is the meter of setting It calculates unit center end and distributes port；
(3) central server sends computing unit data packet and the port numbers for the port for distributing to computing unit center-side To multiple node servers；
(4) after node server receives computing unit data packet, corresponding computing unit node side is set；
(5) node server is the computing unit node side being arranged and computing unit center by the corresponding port of port numbers Connection is established at end.
It is single according to computing unit addition instruction setting calculating is obtained to can be seen that central server by above description Member, and the process for adding computing unit is simple, convenient and fast, it is flexible and convenient to use, and after adding new computing unit, The processing time of video image task can be reduced and task can be handled in time.
As video handles the development with video image processing technology, existing setting in distributed computing resource distribution system Some computing units handled by video image task it is fewer and fewer or even some computing units have been no longer used, still These computing units being no longer used still can occupy central server and the resource of node server causes server resource Waste, for the resource of reasonable employment central server and node server, task processing method is further comprising the steps of (1) To step (4):
(1) central server obtains computing unit and deletes instruction, and computing unit is deleted in instruction and carries computing unit Mark；
(2) central server deletes the mark of the computing unit carried in instruction according to computing unit, unloads computing unit The corresponding computing unit center-side of mark；
(3) central server sends computing unit to multiple node servers and deletes instruction.
(4) when receiving computing unit deletion instruction, node server is according to the calculating in computing unit delete command The mark of unit unloads the corresponding computing unit node side of mark of computing unit.
By above description as can be seen that for not in the computing unit used, the needs that can be handled according to task It is deleted, it is flexible and convenient to use, and also unloading computing unit is convenient and efficient, so that in central server and node server Resource can will not influence other processing units to the treatment process of task by reasonable utilization.
Task processing method provided in this embodiment receives the video image that user sends by the central server of setting Task leads to video image task according to the current task quantity of node servers multiple in server cluster and loading condition Cross central server setting computing unit center-side distribute in server cluster in any node server with the calculating list In the corresponding computing unit node side of first center-side, by the corresponding computing unit node side of video image task to video image Task is handled；By the way that the computing unit center-side being allocated to video image task is arranged in central server, with And setting can handle the computing unit node side of respective type video image task in each server of server cluster, So as to use multiple servers an of server cluster that can handle different types of video image task, it is not necessarily to portion The process demand that multiple server clusters are just able to satisfy video image task is affixed one's name to, the utilization rate of server cluster is improved, is reduced The construction cost of server cluster, and pass through the multiple server process different types of video figure of a server cluster As task, the utilization rate of server resource in server cluster is improved, the waste of resource is avoided.
The computer program product of task processing method provided by the embodiment of the present invention, including storing program code Computer readable storage medium, the instruction that program code includes can be used for executing the method in previous methods embodiment, specific real Now reference can be made to embodiment of the method, details are not described herein.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description, The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed systems, devices and methods, it can be with It realizes by another way.The apparatus embodiments described above are merely exemplary, for example, the division of unit, only For a kind of logical function partition, there may be another division manner in actual implementation.In addition, in each embodiment of the present invention Each functional unit can integrate in one processing unit, is also possible to each unit and physically exists alone, can also two or More than two units are integrated in one unit.
If function is realized in the form of SFU software functional unit and when sold or used as an independent product, can store In a computer readable storage medium.Based on this understanding, technical solution of the present invention is substantially in other words to existing Having the part for the part or the technical solution that technology contributes can be embodied in the form of software products, the computer Software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be personal meter Calculation machine, server or network equipment etc.) execute all or part of the steps of each embodiment method of the present invention.And it is above-mentioned Storage medium includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory The various media that can store program code such as (RAM, Random Access Memory), magnetic or disk.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any Those familiar with the art in the technical scope disclosed by the present invention, can easily think of the change or the replacement, and should all contain Lid is within protection scope of the present invention.Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (10)

1. a kind of distributed computing resource distribution system characterized by comprising central server and multiple node servers, Wherein, the computing unit center-side for different task type, each node serve are provided in the central server Device is respectively arranged with computing unit node side corresponding with the computing unit center-side, the computing unit center-side and correspondence The computing unit node side form a computing unit；
The central server is used to receive the video image task of user's transmission, according to the type of the video image task, And task quantity and loading condition that the multiple node server is current, the video image task is passed through into computing unit Center-side distributes to the corresponding computing unit node side of computing unit center-side described in the node server；
The node server is used for through the corresponding computing unit node side of the video image task to the video image Task is handled；
Wherein, the central server includes: task quantity determining module, for being taken according to node each in current server cluster The task quantity of distribution of business device, determination have distributed task quantity least node server；First choice module, for selecting The least node server of task quantity is as the corresponding node server of the video image task；Second selecting module is used In when the task quantity of distribution for there are multiple node servers is minimum, the utilization of resources in the multiple node server is selected For the minimum node server of rate as the corresponding node server of the video image task, the resource utilization includes following The utilization rate of at least one server hardware resource: central processing unit, memory and network bandwidth；
The node server includes: information reporting module, and current resource is fed back to the central server for periodicity Utilization rate.
2. distributed computing resource distribution system according to claim 1, which is characterized in that the central server packet It includes: task adding module, for searching the task list of the corresponding node server of the video image task, in the task The video image task is added in the task quantity of distribution of list；Task removing module receives the calculating for working as When the completed information of the video image task that cell node end is sent, by the task quantity of distribution of the task list Described in video image task delete；
The node server includes: task feedback module, for passing through the video when the video image task is completed The corresponding computing unit node side of image task sends the completed information of task to the central server, and the task is complete At information carry corresponding computing unit where node server mark.
3. distributed computing resource distribution system according to claim 1, which is characterized in that the central server packet Include: instruction acquisition module, for obtaining computing unit addition instruction, the computing unit addition instruction carries computing unit number According to packet；Center-side setup module, for corresponding computing unit center-side being arranged, and be according to the computing unit data packet The computing unit center-side distribution port being arranged；Sending module, for the computing unit data packet and institute will to be distributed to The port numbers for stating the port of computing unit center-side are sent to the multiple node server；
The node server includes: node side setup module, and after receiving the computing unit data packet, setting is corresponded to Computing unit node side；Connection establishment module, for being the calculating list of setting by the corresponding port of the port numbers First node side and the computing unit center-side establish connection.
4. distributed computing resource distribution system according to claim 1, which is characterized in that the central server packet It includes: deleting instruction acquisition module, delete instruction for obtaining computing unit, the computing unit is deleted in instruction and carries calculating The mark of unit；Computing unit center-side Unload module, for deleting the calculating list carried in instruction according to the computing unit The mark of member, unloads the corresponding computing unit center-side of mark of computing unit；It deletes instruction and issues module, be used for described more A node server sends the computing unit and deletes instruction；
The node server includes: computing unit node side Unload module, is referred to for working as to receive the computing unit and delete When enabling, according to the mark of the computing unit in the computing unit delete command, the mark for unloading the computing unit is corresponding Computing unit node side.
5. distributed computing resource distribution system according to claim 1, which is characterized in that the central server packet Include: common tasks interface obtains the video with different task type that user sends by the common tasks interface Image task.
6. a kind of task processing method realized using system described in any one of claim 1 to 5 characterized by comprising
The central server receives the video image task that user sends, according to the type of the video image task, and The video image task is passed through calculating by the current video image task quantity of the multiple node server and loading condition Distribute to the corresponding computing unit node side of computing unit center-side described in the node server in unit center end；
After the node server receives the video image task, pass through the corresponding computing unit of the video image task Node side handles the video image task；
Wherein, the central server is current according to the type of the video image task and the multiple node server Video image task quantity and loading condition, the video image task is distributed to by computing unit center-side corresponding Node server includes:
The central server is determined and has been distributed according to the task quantity of distribution of node server each in current server cluster The least node server of task quantity；
The central server selects the least node server of task quantity as the corresponding node of the video image task Server；
When the task quantity of distribution for having multiple node servers is minimum, the central server selects the multiple node The minimum node server of resource utilization is as the corresponding node server of the video image task, the money in server Source utilization rate comprises at least one of the following the utilization rate of server hardware resource: central processing unit, memory and network bandwidth；Its In, the node server periodically feeds back current resource utilization to the central server.
7. task processing method according to claim 6, which is characterized in that the method also includes:
The central server searches the task list of the corresponding node server of the video image task, arranges in the task The video image task is added in the task quantity of distribution of table；
When receiving the completed information of the video image task that the computing unit node side is sent, it is described in it is genuinely convinced Video image task described in the task quantity of distribution of the device by the task list of being engaged in is deleted；
When the video image task is completed, the node server passes through the corresponding computing unit of the video image task Node side sends the completed information of task to the central server, and the completed information of task carries corresponding meter The mark of node server where calculating unit.
8. task processing method according to claim 6, which is characterized in that the method also includes:
The central server obtains computing unit addition instruction, and the computing unit addition instruction carries computing unit data Packet；
Corresponding computing unit center-side is arranged according to the computing unit data packet in the central server, and is setting The computing unit center-side distributes port；
The central server is by the port of the computing unit data packet and the port for distributing to the computing unit center-side Number it is sent to the multiple node server；
After the node server receives the computing unit data packet, corresponding computing unit node side is set；
The node server passes through the computing unit node side and the meter that the corresponding port of the port numbers is setting It calculates unit center end and establishes connection.
9. task processing method according to claim 6, which is characterized in that the method also includes:
The central server obtains computing unit and deletes instruction, and the computing unit is deleted in instruction and carries computing unit Mark；
The central server deletes the mark of the computing unit carried in instruction according to the computing unit, unloads the calculating The corresponding computing unit center-side of the mark of unit；
The central server sends the computing unit to the multiple node server and deletes instruction；
When receiving the computing unit deletion instruction, the node server is according in the computing unit delete command The mark of computing unit unloads the corresponding computing unit node side of mark of the computing unit.
10. task processing method according to claim 6, which is characterized in that the central server passes through described general Task interface obtains the video image task with different task type that user sends.
CN201510544350.9A 2015-08-28 2015-08-28 Distributed computing resource distribution system and task processing method Active CN105049268B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510544350.9A CN105049268B (en) 2015-08-28 2015-08-28 Distributed computing resource distribution system and task processing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510544350.9A CN105049268B (en) 2015-08-28 2015-08-28 Distributed computing resource distribution system and task processing method

Publications (2)

Publication Number Publication Date
CN105049268A CN105049268A (en) 2015-11-11
CN105049268B true CN105049268B (en) 2018-12-28

Family

ID=54455490

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510544350.9A Active CN105049268B (en) 2015-08-28 2015-08-28 Distributed computing resource distribution system and task processing method

Country Status (1)

Country Link
CN (1) CN105049268B (en)

Families Citing this family (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106131185B (en) * 2016-07-13 2020-03-17 腾讯科技（深圳）有限公司 Video data processing method, device and system
US10223604B2 (en) * 2016-11-06 2019-03-05 Microsoft Technology Licensing, Llc Live video analytics at scale
CN108062243B (en) * 2016-11-08 2022-01-04 杭州海康威视数字技术股份有限公司 Execution plan generation method, task execution method and device
CN108229260B (en) * 2016-12-21 2020-12-29 杭州海康威视系统技术有限公司 Identity information verification method and system
CN106815070B (en) * 2016-12-30 2020-06-19 北京哲源科技有限责任公司 High-performance computing framework method and system for image analysis
CN106897149A (en) * 2017-03-01 2017-06-27 深圳市博信诺达经贸咨询有限公司 Process the method and system of big data
CN107015767B (en) * 2017-04-06 2020-06-23 南京三宝弘正视觉科技有限公司 NAS device, distributed processing system and method
CN107145384A (en) * 2017-04-17 2017-09-08 广州孩教圈信息科技股份有限公司 Method for allocating tasks and system
CN107241577B (en) * 2017-07-03 2019-08-13 华中科技大学 A kind of processing system for video based on collaborative group mechanism
CN107329834A (en) * 2017-07-04 2017-11-07 北京百度网讯科技有限公司 Method and apparatus for performing calculating task
CN107343045B (en) * 2017-07-04 2021-03-19 北京百度网讯科技有限公司 Cloud computing system and cloud computing method and device for controlling server
CN107622117B (en) * 2017-09-15 2020-05-12 Oppo广东移动通信有限公司 Image processing method and apparatus, computer device, computer-readable storage medium
CN107832150B (en) * 2017-11-07 2021-03-16 清华大学 Dynamic partitioning strategy for computing task
CN107864215B (en) * 2017-11-21 2020-10-20 中国科学院上海微系统与信息技术研究所 Peer-to-peer network file system, access control/management method/system, and terminal
CN107968836B (en) * 2017-12-06 2020-12-18 北京微网通联股份有限公司 Task distribution method and device
CN108023957A (en) * 2017-12-07 2018-05-11 温州中壹技术服务有限公司 A kind of collaborative computer network management system for the processing of information Quick Acquisition
CN110489224A (en) * 2018-05-15 2019-11-22 北京京东尚科信息技术有限公司 A kind of method and apparatus of task schedule
CN109144690B (en) * 2018-07-06 2021-06-22 麒麟合盛网络技术股份有限公司 Task processing method and device
CN109194919A (en) * 2018-09-19 2019-01-11 图普科技（广州）有限公司 A kind of camera data flow distribution system, method and its computer storage medium
CN109670932B (en) * 2018-09-25 2024-02-20 平安科技（深圳）有限公司 Credit data accounting method, apparatus, system and computer storage medium
CN109615869A (en) * 2018-12-29 2019-04-12 重庆集诚汽车电子有限责任公司 Distributed locomotive real-time intelligent is violating the regulations to capture reporting system
CN109714439A (en) * 2019-02-25 2019-05-03 网宿科技股份有限公司 Data processing method and system based on edge calculations
CN109639840A (en) * 2019-02-25 2019-04-16 网宿科技股份有限公司 A kind of data processing method and edge calculations system based on edge calculations
CN111831425B (en) * 2019-04-18 2024-07-16 阿里巴巴集团控股有限公司 Data processing method, device and equipment
CN110308944B (en) * 2019-05-15 2022-03-22 广东电网有限责任公司广州供电局 Configuration file processing method, system, computer device and storage medium
CN110209496B (en) * 2019-05-20 2022-05-17 中国平安财产保险股份有限公司 Task fragmentation method and device based on data processing and fragmentation server
CN110460682A (en) * 2019-09-18 2019-11-15 深圳市网心科技有限公司 A kind of data processing method, device, system and storage medium
CN110764892A (en) * 2019-10-22 2020-02-07 北京字节跳动网络技术有限公司 Task processing method, device and computer readable storage medium
CN110941493A (en) * 2019-11-21 2020-03-31 中国联合网络通信集团有限公司 Task scheduling method and device
CN111212264B (en) * 2019-12-27 2021-08-17 中移（杭州）信息技术有限公司 Image processing method and device based on edge calculation and storage medium
CN111898546B (en) * 2020-07-31 2022-02-18 深圳市商汤科技有限公司 Data processing method and device, electronic equipment and storage medium
CN112905350A (en) * 2021-03-22 2021-06-04 北京市商汤科技开发有限公司 Task scheduling method and device, electronic equipment and storage medium
CN113176940A (en) * 2021-03-29 2021-07-27 新华三信息安全技术有限公司 Data flow splitting method and device and network equipment
CN114968507B (en) * 2021-04-27 2023-08-15 中移互联网有限公司 Image processing task scheduling method and device
CN113342665B (en) * 2021-06-17 2023-10-20 北京百度网讯科技有限公司 Task allocation method and device, electronic equipment and computer readable medium
CN113393367B (en) * 2021-07-08 2022-06-03 北京百度网讯科技有限公司 Image processing method, apparatus, device and medium
CN113590335B (en) * 2021-08-11 2023-11-17 南京大学 Task load balancing method based on grouping and delay estimation in tree edge network
CN114554079B (en) * 2022-01-11 2024-08-06 浙江大华技术股份有限公司 Intelligent service management method and intelligent service management system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103516807A (en) * 2013-10-14 2014-01-15 中国联合网络通信集团有限公司 Cloud computing platform server load balancing system and method
CN103677994A (en) * 2012-09-19 2014-03-26 中国银联股份有限公司 Distributed data processing system, device and method
CN103941662A (en) * 2014-03-19 2014-07-23 华存数据信息技术有限公司 Task scheduling system and method based on cloud computing
CN104850576A (en) * 2015-03-02 2015-08-19 武汉烽火众智数字技术有限责任公司 Fast characteristic extraction system based on mass videos

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103677994A (en) * 2012-09-19 2014-03-26 中国银联股份有限公司 Distributed data processing system, device and method
CN103516807A (en) * 2013-10-14 2014-01-15 中国联合网络通信集团有限公司 Cloud computing platform server load balancing system and method
CN103941662A (en) * 2014-03-19 2014-07-23 华存数据信息技术有限公司 Task scheduling system and method based on cloud computing
CN104850576A (en) * 2015-03-02 2015-08-19 武汉烽火众智数字技术有限责任公司 Fast characteristic extraction system based on mass videos

Also Published As

Publication number Publication date
CN105049268A (en) 2015-11-11

Similar Documents

Publication Publication Date Title
CN105049268B (en) Distributed computing resource distribution system and task processing method
US11327932B2 (en) Autonomous multitenant database cloud service framework
CN104618693B (en) A kind of monitor video based on cloud computing handles task management method and system online
EP3400535B1 (en) System and method for distributed resource management
CN106534318B (en) A kind of OpenStack cloud platform resource dynamic scheduling system and method based on flow compatibility
US8949847B2 (en) Apparatus and method for managing resources in cluster computing environment
CN103152393B (en) A kind of charging method of cloud computing and charge system
CN104050042B (en) The resource allocation methods and device of ETL operations
CN103092698A (en) System and method of cloud computing application automatic deployment
Tao et al. Dynamic resource allocation algorithm for container-based service computing
CN107291536B (en) Application task flow scheduling method in cloud computing environment
CN108920153A (en) A kind of Docker container dynamic dispatching method based on load estimation
WO2016025924A1 (en) Systems and methods for auto-scaling a big data system
CN102880503A (en) Data analysis system and data analysis method
CN110221920B (en) Deployment method, device, storage medium and system
CN105786603B (en) Distributed high-concurrency service processing system and method
CN107370796A (en) A kind of intelligent learning system based on Hyper TF
CN104021029B (en) Spatial information cloud computing system and implementing method thereof
CN111459641B (en) Method and device for task scheduling and task processing across machine room
CN110719320B (en) Method and equipment for generating public cloud configuration adjustment information
CN107402926A (en) A kind of querying method and query facility
CN109614227A (en) Task resource concocting method, device, electronic equipment and computer-readable medium
CN114666333A (en) Control method for cloud computing resource scheduling problem based on multi-tenant theory
CN110661842A (en) Resource scheduling management method, electronic equipment and storage medium
Kijsipongse et al. A hybrid GPU cluster and volunteer computing platform for scalable deep learning

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
PP01 Preservation of patent right
PP01 Preservation of patent right

Effective date of registration: 20220726

Granted publication date: 20181228