New! View global litigation for patent families

WO2014038927A1 - Load balancing of graphics processing units in an image processing system - Google Patents

Load balancing of graphics processing units in an image processing system

Info

Publication number
WO2014038927A1
WO2014038927A1 PCT/MY2013/000165 MY2013000165W WO2014038927A1 WO 2014038927 A1 WO2014038927 A1 WO 2014038927A1 MY 2013000165 W MY2013000165 W MY 2013000165W WO 2014038927 A1 WO2014038927 A1 WO 2014038927A1
Authority
WO
Grant status
Application
Patent type
Prior art keywords
processing
load
buffer
component
gpu
Prior art date
Application number
PCT/MY2013/000165
Other languages
French (fr)
Inventor
Mohanavelu Mudaliar A/L SENAPEN
Oon Leong LIM
Ettikan Kandasamy A/L KARUPPIAH
Hong Hoe ONG
Saw Meng SOO
Original Assignee
Mimos Berhad
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed circuit television systems, i.e. systems in which the signal is not broadcast
    • H04N7/183Closed circuit television systems, i.e. systems in which the signal is not broadcast for receiving images from a single remote source
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/232Devices for controlling television cameras, e.g. remote control; Control of cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in, e.g. mobile phones, computers or vehicles
    • H04N5/23203Remote control signaling for cameras or for parts of camera, e.g. between main body and part of camera
    • H04N5/23206Remote control signaling for cameras or for parts of camera, e.g. between main body and part of camera using a network, e.g. internet
    • GPHYSICS
    • G08SIGNALLING
    • G08BSIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B13/00Burglar, theft or intruder alarms
    • G08B13/18Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength
    • G08B13/189Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems
    • G08B13/194Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
    • G08B13/196Actuation by interference with heat, light or radiation of shorter wavelength; Actuation by intruding sources of heat, light or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
    • G08B13/19665Details related to the storage of video surveillance data
    • G08B13/19667Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements

Abstract

An image processing system (2) comprising at least one imaging source (4) for capturing images and/or video, one or more graphics processing units (28) for processing the images and/or video, at least one monitoring component (26) for monitoring the processing load of the graphics processing units (28), and a load adjuster component (30) for adjusting the processing load of the graphics processing units (28) by increasing or reducing the rate at which images and/or video are captured by the imaging source (4).

Description

LOAD BALANCING OF GRAPHICS PROCESSING UNITS IN AN IMAGE

PROCESSING SYSTEM

Field of Invention The invention relates to load balancing of graphics processing units in an image processing system.

Background

In a conventional system for processing video, the frames thereof may be distributed by a controller to a plurality of graphics processing unit (GPU) for video analytics (VA) such as fire detection, motion detection, theft, etc.

However, if the processing load is too high for a particular GPU, a feedback signal is sent from that GPU to the controller to temporarily inhibit further transfer of frames thereto from the controller.

Unfortunately, this can result in frames being dropped when they are not processed, which reduces accuracy due to non-smooth transitions. An aim of the invention is to address the aforementioned issue. Summary of Invention

In an aspect of the invention, there is provided an image processing system comprising:

at least one imaging source for capturing images and/or video;

one or more graphics processing units for processing the images and/or video; at least one monitoring component for monitoring the processing load of the graphics processing units; and

a load adjuster component for adjusting the processing load of the graphics processing units;

characterised in that the load adjuster component adjusts the processing load by increasing or reducing the rate at which images and/or video are captured by the imaging source. Thus by adjusting the image capture rate at source, overloading of the graphics processing units is minimised, resulting in less frames being dropped and smooth frame transitions.

In one embodiment the imaging source is a camera.

In one embodiment if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component reduces the capture rate of the imaging source by one frame per second. In one embodiment if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second.

In one embodiment configuration information for the imaging source is read from an XML configuration file. Typically the initial frame rate of the imaging source is set at the imaging source's maximum capture rate e.g. 25 frames per second. Typically the minimum frame rate is set at the minimum rate for effective video analytics e.g. 6 frames per second. It will be appreciated that any suitable rates could be applied depending on the requirements.

In one embodiment the system includes a server in which the graphics processing units, monitoring component and load adjuster component are located. Typically the images and/or video are transmitted to the server via a network such as the internet.

In one embodiment the processed images and/or video are transmitted to a client for viewing, typically via a network such as the internet. In one embodiment the images and/or video are received by the server, decoded, and stored in a temporary buffer before they are processed by a graphics processing unit. In one embodiment if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the temporary buffer. In one embodiment the images and/or video are stored in a network buffer before being transmitted to the server.

In one embodiment if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the network buffer. Thus network bandwidth is saved as the frames may be discarded prior to transmission.

In one embodiment a processing component is provided to determine which images and/or video are to be processed by a graphics processing unit.

In one embodiment the images and/or video are passed to a graphics processing unit for video analytics processing, typically based on image timestamp.

In one embodiment the monitoring component includes a timer to measure the number of frames processed per second.

In a further aspect of the invention, there is provided a method of load balancing the processing of images and/or video in an image processing system comprising: capturing images and/or video via at least one imaging source; processing the images and/or video via a plurality of graphics processing units; monitoring the processing load of the graphics processing units via at least one monitoring component for; and adjusting the processing load of the graphics processing units via a load adjuster component; characterised in that the processing load is adjusted by the load adjuster component by increasing or reducing the rate at which images and/or video are captured by the imaging source.

Brief Description of Drawings

It will be convenient to further describe the present invention with respect to the accompanying drawings that illustrate possible arrangements of the invention. Other arrangements of the invention are possible, and consequently the particularity of the accompanying drawings is not to be understood as superseding the generality of the preceding description of the invention.

Figure 1 illustrates a schematic overview of the system according to an embodiment of the invention.

Figure 2 illustrates a system architecture block diagram according to an embodiment of the invention. Figure 3 illustrates a block diagram of the image acquisition and storage process. Figure 4 illustrates a block diagram of the data buffering process.

Figure 5 illustrates a block diagram of the processing component initialization sequence.

Figure 6 illustrates a block diagram of the load adjuster module.

Figure 7 illustrates a block diagram of the GPU monitoring component.

Figure 8 illustrates a block diagram of the frame fetching and dropping mechanism. Detailed Description

With reference to Figure 1 there is illustrated the overall system 2 consisting of an imaging source in the form of a camera 4, a server 6 which has a central processing unit (CPU) 8, graphics processing unit (GPU) 10 and a network card 12, and viewing station 14 which receives processed images from the server 6. The CPU 8 manages the whole system, the GPU is used for image processing 10 and the network card 12 is used to send processed images to a viewing client 16 on the viewing station 14 and receive the images from the camera 4, typically via the internet 18. Figure 2 depicts the system architecture block diagram, comprising a camera communication module 20 which receives images from the camera 4, a buffer management module 22 for temporarily storing images, a processing component 24 for determining which received images to process, a GPU monitoring thread 26 for monitoring GPU performance, a GPU card 28 for performing the image processing and streaming, a GPU processing load adjustment module 30, a client processing module 32, and a client communication module 34 for output to a viewing client 16 on the viewing station 14.

The GPU processing load adjuster module 30 is responsible for adjusting the input load to the GPU 28. If the load adjuster module 30 finds the GPU 28 to be busy, it will reduce the camera acquisition rate by 1, drops frame at the network data buffer i.e. drops packets before decoding, and also drops frames at the temporary storage buffer which stores decoded images. If the GPU is free, the camera frame rate will be increased and each frame will be processed without frame drops.

The GPU processing load adjuster component 30 interacts with the camera 4, camera communication module 20 and buffer management module 22 for load adjustment. The processing component 24 will pass the images to the GPU 28 for Video Analytics (VA) processing 34. The GPU monitoring component 26 is an object with an independent thread 36 monitoring the GPU device output buffer to determine GPU load. It provides feedback of GPU status 38 which is picked up by the Load Adjuster module 30. The camera communication module 20 connects 40 to the camera 4 and receives images 42. The images are decoded 44 based on received format e.g. MPEG or MJPEG and a temporary buffer is then checked 46 to determine 48 if it is full or not. If not full the frame is stored in the temporary buffer 50. If full the frame may be moved to a discard buffer 52 or the latest frame may be pushed into the temporary buffer 50. The discard buffer 52 does not immediately discard the frames, but retains them in round robin fashion.

The decoded image will be fetched 54 by image data processing module 24 once the GPU is free for VA processing. Data availability is checked 56, and if available is sent 58 to the GPU 28 for processing 34. If data is not available the GPU status is checked 60 via the Load Adjuster module 30 to determine if it is busy or not 62. If the GPU is not busy, data is fetched 54 by the image data processing module 24. If the GPU is busy, the frame may be moved to the discard buffer 52. The camera acquisition rate may also be adjusted 64 accordingly.

Thus if the GPU is busy 62, the frames in the temporary buffer 50 and discard buffer 52 need to be in sync based on time stamp, and the latest frame from the discard buffer 52 may be pushed back to the temporary buffer 50 accordingly. Older frames in the discard buffer 50 can be deleted.

For the client 16 the streaming component through client communication module 34 will transmit and receive requests 66 for images from the view station 14. Data is read 68 from the GPU 28 and stored 70 in the processed data buffer. The data is fetched 72 by a specific thread and if available 74 is sent to the viewing client 16.

Figure 3 illustrates further details of the image acquisition and storage process. During initialization 302 the configuration information such as camera IP address, login and password are read 304 from the xml configuration file 306. The configuration information is passed to an acquisition component 308 and the connection to the camera is initialized through a web-url mechanism 310. The initial frame rate is set 312 at a maximum frame rate such as 25 fps. Thus as soon as the image is captured at the source device in this case is an IP cam, images will be streamed into acquisition component 314 at this rate. At this stage, the data is stored 316 at the network buffer with timestamp information before being passed into a decoding buffer. The image is decoded 44 based on the acquired image format such as MJPEG or MPEG format, time stamped 318, then stored in the temporary buffer 50. Figure 4 shows data buffering details. The network buffer 404, decoding buffer 406 and processing buffer 408 are each managed in the same way. Once data is received 410 the next available slot is determined 412 to store the data. The buffer limit is checked 414, and if the limit is reached the pointer will be reset 416 to start position, otherwise it points to the next available buffer slot 418. The current available slot number is returned 420 so that data can be stored at that location 422. Thus the buffers are managed in ring buffer fashion based on FIFO queues. It is assumed that data will be arriving in sequence as it is transmitted using a TCP/IP framework. Figure 5 shows the initialization sequence of the processing component 24. This starts 502 by reading 504 the same xml configuration file 306 as described above, as the load adjustment module 30 requires camera details to explicitly set the frame rate based on GPU load. Thus the configuration information is used to initialise 506 the processing component before initialising the load adjuster module 30 and the GPU monitoring thread 26.

Figure 6 shows the overall flow of the load adjuster module 30. After initialisation 602, load adjuster module operation 604 is determined by a GPU status flag 606, which is controlled by the GPU monitoring module 26.

The status of the GPU flag is determined 608, and if found to be busy, the details of the current image source device are acquired 610 and the current frame rate of this device is obtained 612. If the frame rate is determined 614 to be above a threshold e.g. 6, the frame rate is decremented 616 by 1 and a new frame rate is explicitly set 618 at the device and stored locally 620. In addition, the communication module 20 and the buffer management module 22 are each respectively signalled 622, 624 to drop the frame.

If the GPU is free, data for image processing operation using the GPU will be fetched 626, if available 628, from the buffer management module 22. Once data is fetched it will be sent 630 to the GPU 632 for processing. The processed data is read 634 from the GPU, the processed frame count is increased 636, and the processed data is stored 638 in the processed buffer, so that it can be fetched later by the streaming component. In addition, the acquisition device is identified 640 and the acquisition rate is increased 642 by 1. The maximum rate is taken care of by the device itself and no frames are dropped at the acquisition side. The GPU status is continued to be monitored 644, wherein the frame count is used to measure the processing rate by the GPU monitoring component 26 to determine if the GPU status is busy or free.

Figure 7 shows the GPU monitoring component operation. The operation starts 702 and the GPU monitoring thread is initialised 704 wherein a timer is set 706 to activate timer code each second thus measuring the rate of operation. When the timer code is activated 708, the current processed frame count of the device is acquired 710 and measured against the previous frame count 712, thereby providing a figure for how many frames are processed within a second. Processed frame count is always incremented to keep track of how many frames are processed. Overflow is taken care of during a comparison operation, which is required because the GPU monitoring thread is an independent thread from the processing thread.

The processed frame count is checked 714, and the current GPU is checked 716 to determine its status 718. If the frame count exceeds the threshold value e.g. 6, for example because the GPU is processing the frames quickly, but the GPU status flag is set to busy, then the GPU status flag is set to free 720. If the frame count is below the threshold value i.e. the GPU is processing frames slowly due to heavy load, and the GPU status flag is set to free, then the GPU status flag is set to busy 722. If the frame count exceeds the threshold value and the GPU status is free, or the frame count is below the threshold value and the GPU is busy, the GPU status flag is not changed.

The previous frame count is updated 724 to the current frame count and the code ends 726.

Figure 8 shows the frame fetching and dropping mechanism. Frame fetching and dropping mechanism is the same at the network buffer 404, buffer management module 22 and also at the streaming side via client processing module 32. The network communication and buffer management module handler 802 comprises a buffer data fetch handler, a frame drop handler, and a buffer store data handler. Basically frames are fetched based on position of frame fetch pointer once there is a request to get current frame. Buffer data Fetch handler 804 will receive a request to fetch 44, 54, 72 data for decoding (at network side), processing, or streaming respectively. Data is returned 806 based on the current location of the fetch data pointer which acts as an indicator for respective data. If the fetch Pointer reaches the end of the buffer 808 after the current data is returned, the pointer is shifted 810 to the first location i.e. the header of the buffer, else the current location is incremented 812 to point to the next location.

The frame drop handler 814 receives requests 816 to drop frames, and in such circumstances the fetch pointer is incremented by one, thus skipping one frame. The request to discard a frame for network can be received from gpu processing load adjuster 30, processing 62, or streaming handled within the client communication module 34 (so as to ensure old frames are discarded if the client is down). Once notification is received, if the current fetch pointer for frame dropping is pointing at the end of buffer, one frame is skipped by resetting the pointer to head/start location else one frame is dropped by incrementing the current pointer. There will be no discard requests from the processed data buffer, as processed frames are stored 70 in round robin fashion. With regard to the buffer store data handler 818, data is stored at the buffer based on store pointer at the next available slot. A request to store data at a particular location for network can be received 820 from the camera 4, the buffer management module 22 (temporary buffer 50 or discard buffer 52) and for streaming (storing 70 in the processed data buffer). Once data is received, the location to store the data needs to be checked 822. If the location pointer is pointing to the end of buffer, it needs to be reset 824 to the head/start location else the current pointer is incremented 826 to point to the next empty slot. Data is then stored 828 at this location.

It will be appreciated by persons skilled in the art that the present invention may also include further additional modifications made to the device which does not affect the overall functioning of the device.

Claims

Claims
An image processing system (2) comprising
at least one imaging source (4) for capturing images and/or video;
one or more graphics processing units (28) for processing the images and/or video;
at least one monitoring component (26) for monitoring the processing load of the graphics processing units (28); and
a load adjuster component (30) for adjusting the processing load of the graphics processing units (28);
characterised in that the load adjuster component (30) adjusts the processing load by increasing or reducing the rate at which images and/or video are captured by the imaging source (4).
An image processing system according to claim 1 wherein if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component reduces the capture rate of the imaging source by one frame per second. 3. An image processing system according to claim 1 wherein if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second. An image processing system according to claim 1 wherein the images and/or video are transmitted to a server in which the graphics processing units, monitoring component and load adjuster component are located.
An image processing system according to claim 1 wherein the processed images and/or video are transmitted to a client for viewing.
An image processing system according to claim 1 wherein the images and/or video are decoded and stored in a temporary buffer before they are processed by a graphics processing unit.
An image processing system according to claim 6 wherein if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the temporary buffer.
An image processing system according to claim 1 wherein the images and/or video are stored in a network buffer before being transmitted from the imaging source, and if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the network buffer.
9. An image processing system according to claim 1 wherein images and/or video are passed to a graphics processing unit for video analytics processing based on image timestamp. 10. A method of load balancing the processing of images and/or video in an image processing system (2) comprising:
capturing images and/or video via at least one imaging source (4);
processing the images and/or video via a plurality of graphics processing units (28);
monitoring the processing load of the graphics processing units via at least one monitoring component (26); and
adjusting the processing load of the graphics processing units via a load adjuster component (30);
characterised in that the processing load is adjusted by the load adjuster component (30) by increasing or reducing the rate at which images and/or video are captured by the imaging source (4).
1 1. A method according to claim 10 wherein if the monitoring component detects that a graphics processing unit is busy, the load adjuster component reduces the capture rate of the imaging source by one frame per second and/or discards images and/or video frames received therefrom.
12. A method according to claim 10 wherein if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second.
PCT/MY2013/000165 2012-09-10 2013-09-09 Load balancing of graphics processing units in an image processing system WO2014038927A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
MYPI2012004012 2012-09-10
MYPI2012004012 2012-09-10

Publications (1)

Publication Number Publication Date
WO2014038927A1 true true WO2014038927A1 (en) 2014-03-13

Family

ID=49356484

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/MY2013/000165 WO2014038927A1 (en) 2012-09-10 2013-09-09 Load balancing of graphics processing units in an image processing system

Country Status (1)

Country Link
WO (1) WO2014038927A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016015667A1 (en) * 2014-08-01 2016-02-04 广州金山网络科技有限公司 Video stream decoding method and device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020126751A1 (en) * 1998-05-22 2002-09-12 Christoph E. Scheurich Maintaining a frame rate in a digital imaging system
US20030107648A1 (en) * 2001-12-12 2003-06-12 Richard Stewart Surveillance system and method with adaptive frame rate
EP1936576A1 (en) * 2006-12-20 2008-06-25 Axis AB Camera tampering detection
US20090135918A1 (en) * 2007-11-23 2009-05-28 Research In Motion Limited System and method for providing a variable frame rate and adaptive frame skipping on a mobile device
US20110211036A1 (en) * 2010-02-26 2011-09-01 Bao Tran High definition personal computer (pc) cam

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020126751A1 (en) * 1998-05-22 2002-09-12 Christoph E. Scheurich Maintaining a frame rate in a digital imaging system
US20030107648A1 (en) * 2001-12-12 2003-06-12 Richard Stewart Surveillance system and method with adaptive frame rate
EP1936576A1 (en) * 2006-12-20 2008-06-25 Axis AB Camera tampering detection
US20090135918A1 (en) * 2007-11-23 2009-05-28 Research In Motion Limited System and method for providing a variable frame rate and adaptive frame skipping on a mobile device
US20110211036A1 (en) * 2010-02-26 2011-09-01 Bao Tran High definition personal computer (pc) cam

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016015667A1 (en) * 2014-08-01 2016-02-04 广州金山网络科技有限公司 Video stream decoding method and device

Similar Documents

Publication Publication Date Title
US7330434B2 (en) Traffic quality measurement apparatus and method
US20100232518A1 (en) System and method for streaming video to a mobile device
US20060078046A1 (en) Method and system for multi-path video delivery and distribution
US20070126875A1 (en) Multi-codec camera system and image acquisition program
US20110115932A1 (en) Method and apparatus for providing image in camera or remote-controller for camera
US7623152B1 (en) High resolution network camera with automatic bandwidth control
US20120265847A1 (en) Real-Time Video Detector
US20130141523A1 (en) Apparatus and Method for Panoramic Video Hosting
US20060242678A1 (en) Remote network video content recorder system
US20060104345A1 (en) Method and apparatus for controlling a video surveillance display
US8265154B2 (en) Redundant data dispersal in transmission of video data based on frame type
US20090310668A1 (en) Method, apparatus and system for concurrent processing of multiple video streams
EP1130839A1 (en) Method and apparatus for retransmitting video data frames with priority levels
US20100135381A1 (en) Encoding/decoding device and video transmission system
US20120307049A1 (en) Networked security camera with local storage and continuous recording loop
US20120230390A1 (en) Adaptive Control of Encoders for Continuous Data Streaming
US20110252265A1 (en) Delay controller, control method, and communication system
US20120324123A1 (en) Method and apparatus for session bandwidth estimation and rate control
US20120250762A1 (en) System and method for implementation of dynamic encoding rates for mobile devices
US20090238542A1 (en) Capturing event information using a digital video camera
US20090135252A1 (en) Monitoring camera device, monitoring system using the same, and monitoring image transmission method
US20090285310A1 (en) Receiving apparatus, receiving method, program and communication system
US20050088519A1 (en) Video surveillance system
US20120300081A1 (en) Surveillance system
US20100011402A1 (en) Communication apparatus and communication method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13776869

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct app. not ent. europ. phase

Ref document number: 13776869

Country of ref document: EP

Kind code of ref document: A1