CN105049485A - Real-time video processing oriented load-aware cloud calculation system - Google Patents
Real-time video processing oriented load-aware cloud calculation system Download PDFInfo
- Publication number
- CN105049485A CN105049485A CN201510330962.8A CN201510330962A CN105049485A CN 105049485 A CN105049485 A CN 105049485A CN 201510330962 A CN201510330962 A CN 201510330962A CN 105049485 A CN105049485 A CN 105049485A
- Authority
- CN
- China
- Prior art keywords
- load
- worker
- cpu
- gpu
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0882—Utilisation of link capacity
Abstract
The invention provides a real-time video processing oriented load-aware cloud calculation system. The system comprises a Storm cluster used for providing infrastructure service, a video streaming generator used for generating, receiving, and sending video streaming, a streaming server used for caching video data, providing a unified message interface, and reducing the coupling between assemblies, a load detector, a parameter controller used for performance analysis and evaluation of the cluster, a video processor used for respectively providing interfaces based on CPU and GPU processing, and a protocol supplier used for providing various protocols, the load detector binds a load awareness algorism to aware the calculating load of a task and notifies the Storm cluster to select the needed processor type via the analysis of CPU, GPU, and memory usage conditions of a node of the task, and the parameter controller interacts with the Storm cluster via the protocol supplier so that message exchange under different protocols is realized, and the coupling between modules is removed.
Description
Technical field
The present invention relates to the large data of cloud computing, field of video processing, be specifically related to a kind of Load-aware cloud computing system towards real time video processing.
Background technology
Flourish along with the business models such as the Internet, mobile device, Smart Home, intelligent transportation and research direction, magnanimity real time data needs to be utilized effectively, and wherein because real-time is high, the features such as data volume is large, video data seems particularly important.The up-to-date report of InternationalDataCorporation " TheDigitalUniversein2020 " is pointed out, the half of the large data in the whole world in 2012 is all video data, will reach 65% to this ratio in 2015.
For CPU, adopt GPU process video greatly can improve computational speed.But current GPU processing platform weak point is, although consume less CPU based on the task of GPU, but can take a large amount of internal memories.Particularly when low memory, this bottleneck can limit treatment effeciency greatly.
Therefore, mass data how is utilized efficiently and fully the information excavated therebetween has become this area problem demanding prompt solution.
Summary of the invention
In order to solve under cloud environment intelligent information between live video stream, to excavate the resource consumption brought large, the problem that data volume is large, the present invention proposes a kind of Load-aware cloud computing system towards real time video processing on the basis based on cloud computing and large data platform, energy maximum using cloud node resource, improves computational speed.
Technical scheme of the present invention is achieved in that
Towards a Load-aware cloud computing system for real time video processing, comprising:
Storm cluster, the service of providing infrastructures;
Video stream generator, for the generation of video flowing, accepts and sends;
Streaming server, buffered video data also provides unified message interface, reduces the coupling between assembly;
Load detector, its binding Load-aware algorithm is with the computational load of perception task, and by CPU, GPU of task place node and the analysis of internal memory service condition, notice Storm cluster should selection processor type;
Parameter controller, for the assessment and analysis of the performance of cluster;
Video processor, it provides the interface based on CPU and GPU process respectively;
Agreement supplier, provides various agreement, and by agreement supplier, described parameter controller and Storm cluster carry out alternately, realizing the message under different agreement, removing the coupling of intermodule simultaneously.
Alternatively, described agreement supplier provides AOP, RMI, http protocol.
Alternatively, first described load dynamic sensing algorithm added up the computational resource of each computing node in cloud environment before topological Hand up homework; After this topology is submitted to, obtain the resource request of each worker of topology, this request is compared with the available resources of each cloud node, when the available resources of certain cloud node are greater than the request of this worker, this worker is planned for this node; Then, by comparing the difference request of this worker to CPU and GPU resource, calculating this worker when using CPU and use GPU respectively and can be received number, selecting the processor that can hold maximum number.
The invention has the beneficial effects as follows:
(1) construct a kind of efficient cloud computing system towards magnanimity real time video processing, achieve the Intelligent treatment to magnanimity real-time video;
(2) propose the load dynamic sensing algorithm that a kind of CPU and GPU combines, achieve the maximum using of cloud node calculate resource.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the theory diagram of the present invention towards the Load-aware cloud computing system of real time video processing;
Fig. 2 is the operational flow diagram of the present invention towards the Load-aware cloud computing system of real time video processing;
Fig. 3 is the load dynamic sensing algorithm flow chart that CPU and GPU of the present invention combines.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
As shown in Figure 1, the present invention comprises towards the Load-aware cloud computing system of real time video processing: Storm cluster, video stream generator, streaming server, load detector, parameter controller, video processor and agreement supplier.
Storm cluster as core component of the present invention, for the service of providing infrastructures.
Video stream generator is responsible for the generation of video flowing, accepts and sends.
Streaming server is used for buffered video data and provides unified message interface, reduces the coupling between assembly.
Load detector binding load dynamic sensing algorithm is with the computational load of perception task.By to CPU, GPU of task place node and the analysis of internal memory service condition, notify that Storm cluster should selection processor type.
Parameter controller is used for the assessment and analysis of the performance of cluster.
Video processor provides the interface based on CPU and GPU process respectively.
Agreement supplier provides various agreement, comprises AOP, RMI, HTTP etc.
Fig. 2 is the operational flow diagram of the present invention towards the Load-aware cloud computing system of real time video processing.
After a new Hand up homework to system of the present invention, first system checks whether Storm platform also has many virtual machine processes to run this operation.
If so, system so of the present invention just meets all worker that this task is asked, and after all worker are assigned with, each worker just starts oneself operation.
It should be noted that some worker can not comprise the heavy work of calculation task, so system of the present invention also comprises the load dynamic sensing algorithm that CPU and GPU combines, first check whether a certain worker comprises video calculation task.If comprised, so just perform Load-aware algorithm, determine this task matching to calculate to CPU or GPU calculating.
After all worker are traversed, whole operation just starts to perform.
Whether affect the trouble-free operation of other operations to detect this operation of uploading new, system of the present invention also can detect the performance of whole Storm platform simultaneously, if whole platform property is affected, just proposes to report to the police.
Different calculation tasks, may be applicable to CPU process, also may be applicable to GPU process, so the present invention proposes the load dynamic sensing algorithm that a kind of CPU and GPU combines, realizes the Coordination Treatment of CPU-GPU.Cloud computing node RAM or CPU, by situation about too much taking, can cause hydraulic performance decline.And in order to avoid this situation, we arrange two threshold value CPU usage α and RAM utilization rate β.As shown in Figure 3, according to the loading condition of node, carry out the algorithm that CPU-GPU selects flexibly and can be defined as follows:
N={n
i}: node set;
TC
i, UC
i: the overall CPU of node i and the CPU of use;
TR
i, UR
i: the point overall RAM of i and the RAM of use;
AC, AR: node can use CPU and RAM;
Topo={t
j}: the set of application deployment distribution map (topologies);
W
j={ w
j, k}: the process (workers) of distributing is needed from a jth figure (topology);
Grc
j, k, grr
j, k: process w
j, kcPU and RAM needed when using GPU;
Crc
j, k, crr
j, k: w
j, kcPU and RAM needed when using CPU.
First the load dynamic sensing algorithm that CPU and GPU of the present invention combines added up the computational resource of each computing node in cloud environment before topological Hand up homework; After this topology is submitted to, obtain the resource request of each worker of topology, this request is compared with the available resources of each cloud node, when the available resources of certain cloud node are greater than the request of this worker, this worker is planned for this node; Then, by comparing the difference request of this worker to cpu and gpu resource, calculating this worker when using cpu and use gpu respectively and can be received number, selecting the processor that can hold maximum number.
Load-aware cloud computing system towards real time video processing of the present invention, achieves the Intelligent treatment to magnanimity real-time video; Propose the load dynamic sensing algorithm that a kind of CPU and GPU combines, achieve the maximum using of cloud node calculate resource.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (3)
1., towards a Load-aware cloud computing system for real time video processing, it is characterized in that, comprising:
Storm cluster, the service of providing infrastructures;
Video stream generator, for the generation of video flowing, accepts and sends;
Streaming server, buffered video data also provides unified message interface, reduces the coupling between assembly;
Load detector, its binding load dynamic sensing algorithm is with the computational load of perception task, and by CPU, GPU of task place node and the analysis of internal memory service condition, notice Storm cluster should selection processor type;
Parameter controller, for the assessment and analysis of the performance of cluster;
Video processor, it provides the interface based on CPU and GPU process respectively;
Agreement supplier, provides various agreement, and by agreement supplier, described parameter controller and Storm cluster carry out alternately, realizing the message under different agreement, removing the coupling of intermodule simultaneously;
After a new Hand up homework to native system, first system checks whether Storm platform also has many virtual machine processes to run this operation; If so, then system meets all worker that this task is asked, and after all worker are assigned with, each worker just starts oneself operation; Described load dynamic sensing algorithm, first checks whether a certain worker comprises video calculation task, if comprised, performs load dynamic sensing algorithm, determines this task matching to calculate to CPU or GPU calculating; After all worker are traversed, whole operation just starts to perform.
2., as claimed in claim 1 towards the Load-aware cloud computing system of real time video processing, it is characterized in that, described agreement supplier provides AOP, RMI, http protocol.
3., as claimed in claim 1 towards the Load-aware cloud computing system of real time video processing, it is characterized in that, described load dynamic sensing algorithm comprises the following steps:
First before topological Hand up homework, add up the computational resource of each computing node in cloud environment;
After this topology is submitted to, obtain the resource request of each worker of topology, the available resources of this request with each cloud node are compared, when the available resources of certain cloud node are greater than the request of this worker, this worker is planned for this node;
Then, by comparing the difference request of this worker to CPU and GPU resource, calculating this worker when using CPU and use GPU respectively and can be received number, selecting the processor that can hold maximum number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510330962.8A CN105049485B (en) | 2015-06-09 | 2015-06-09 | A kind of Load-aware cloud computing system towards real time video processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510330962.8A CN105049485B (en) | 2015-06-09 | 2015-06-09 | A kind of Load-aware cloud computing system towards real time video processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105049485A true CN105049485A (en) | 2015-11-11 |
| CN105049485B CN105049485B (en) | 2018-10-16 |
Family
ID=54455688
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510330962.8A Active CN105049485B (en) | 2015-06-09 | 2015-06-09 | A kind of Load-aware cloud computing system towards real time video processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105049485B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106095573A (en) * | 2016-06-08 | 2016-11-09 | 北京大学 | The Storm platform operations of a kind of work nest perception divides equally dispatching method |
| CN106878671A (en) * | 2016-12-29 | 2017-06-20 | 中国农业大学 | A kind of plant's multiple target video analysis method and its system |
| CN108037995A (en) * | 2017-11-22 | 2018-05-15 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Distributed electromagnetic situation simulation computing system based on GPU |
| CN109857560A (en) * | 2019-01-28 | 2019-06-07 | 中国石油大学(华东) | A kind of collaboration parallelization mechanism based on CPU/GPU isomerous environment |
| CN110035297A (en) * | 2019-03-08 | 2019-07-19 | 视联动力信息技术股份有限公司 | Method for processing video frequency and device |
| CN112346863A (en) * | 2020-10-28 | 2021-02-09 | 河北冀联人力资源服务集团有限公司 | Method and system for processing dynamic adjustment data of computing resources |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1677190A2 (en) * | 2004-12-30 | 2006-07-05 | Microsoft Corporation | Systems and methods for virtualizing graphics subsystems |
| CN103699656A (en) * | 2013-12-27 | 2014-04-02 | 同济大学 | GPU-based mass-multimedia-data-oriented MapReduce platform |
| CN104125165A (en) * | 2014-08-18 | 2014-10-29 | 浪潮电子信息产业股份有限公司 | Job scheduling system and method based on heterogeneous cluster |
-
2015
- 2015-06-09 CN CN201510330962.8A patent/CN105049485B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1677190A2 (en) * | 2004-12-30 | 2006-07-05 | Microsoft Corporation | Systems and methods for virtualizing graphics subsystems |
| CN103699656A (en) * | 2013-12-27 | 2014-04-02 | 同济大学 | GPU-based mass-multimedia-data-oriented MapReduce platform |
| CN104125165A (en) * | 2014-08-18 | 2014-10-29 | 浪潮电子信息产业股份有限公司 | Job scheduling system and method based on heterogeneous cluster |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106095573A (en) * | 2016-06-08 | 2016-11-09 | 北京大学 | The Storm platform operations of a kind of work nest perception divides equally dispatching method |
| CN106878671A (en) * | 2016-12-29 | 2017-06-20 | 中国农业大学 | A kind of plant's multiple target video analysis method and its system |
| CN106878671B (en) * | 2016-12-29 | 2019-07-26 | 中国农业大学 | A kind of farm's multiple target video analysis method and its system |
| CN108037995A (en) * | 2017-11-22 | 2018-05-15 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Distributed electromagnetic situation simulation computing system based on GPU |
| CN109857560A (en) * | 2019-01-28 | 2019-06-07 | 中国石油大学(华东) | A kind of collaboration parallelization mechanism based on CPU/GPU isomerous environment |
| CN110035297A (en) * | 2019-03-08 | 2019-07-19 | 视联动力信息技术股份有限公司 | Method for processing video frequency and device |
| CN110035297B (en) * | 2019-03-08 | 2021-05-14 | 视联动力信息技术股份有限公司 | Video processing method and device |
| CN112346863A (en) * | 2020-10-28 | 2021-02-09 | 河北冀联人力资源服务集团有限公司 | Method and system for processing dynamic adjustment data of computing resources |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105049485B (en) | 2018-10-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105049485A (en) | Real-time video processing oriented load-aware cloud calculation system | |
| CN104243481B (en) | A kind of method and system of power information acquisition advance data processing | |
| CN103501347B (en) | Resource allocation methods and device | |
| CN103281366A (en) | Embedded agency monitoring device and method supporting real-time operating state acquiring | |
| CN110493304B (en) | Edge computing system and transaction system | |
| EP3011494A2 (en) | System and method for distributed computation using heterogeneous computing nodes | |
| CN110688213B (en) | Resource management method and system based on edge calculation and electronic equipment | |
| CN103856337A (en) | Resource occupation rate acquiring method, providing method, system and server thereof | |
| Kchaou et al. | Towards an offloading framework based on big data analytics in mobile cloud computing environments | |
| CN105491329A (en) | Large-scale monitoring video stream converging method based on stream-oriented computation | |
| CN104468710A (en) | Mixed big data processing system and method | |
| Saab et al. | Energy efficiency in mobile cloud computing: Total offloading selectively works. does selective offloading totally work? | |
| CN103731484B (en) | A kind of power save transmission method towards mobile cloud computing and middleware system | |
| Ou et al. | Research on network performance optimization technology based on cloud-edge collaborative architecture | |
| CN104796673A (en) | Energy consumption optimization-oriented cloud video monitoring system task access method | |
| CN116402318A (en) | Multi-stage computing power resource distribution method and device for power distribution network and network architecture | |
| CN102565477A (en) | Electric energy meter alarm information transmission system and electric energy meter alarm information uploading method | |
| CN115858672A (en) | Power terminal management method and device, electronic equipment and storage medium | |
| CN106997310A (en) | The apparatus and method of load balancing | |
| Gill et al. | A computation offloading scheme for performance enhancement of smart mobile devices for mobile cloud computing | |
| Al-Rubaie et al. | Simulating fog computing in OMNeT++ | |
| CN114301980A (en) | Method, device and system for scheduling container cluster and computer readable medium | |
| Chen et al. | Cloud-edge collaborative data processing architecture for state assessment of transmission equipments | |
| Li | An integrated web service framework for mobile device hosted web service and its performance analysis | |
| Khanna et al. | Adaptive mobile computation offloading for data stream applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |