CZ307563B6 - A method of dynamic distribution of computing processes to one or more graphics processing units (GPU) - Google Patents

Abstract

GPU manager is a scheduler based on a discrete multi-criteria optimization algorithm that ensures optimal neural network (03) layout on individual graphics processors (09). It schedules on which graphics processors (09) neural networks (03) will run and if multiple copies will run in order to maximize the number of processed images per unit of time. For this purpose, it uses the principles of asynchronous and parallel data processing. It further provides communication between proxy detectors (02) with image streams and instances of neural networks (03), i.e. start, termination, configuration changes and pairing.

Description

A method of dynamically allocating computational processes to one or more graphics processors

Field of technology

The present invention relates to a real-time computer analysis of video from security cameras, based on the dynamic distribution of individual computational processes to one or more graphics cards. The invention enables semantic analysis (object detection) of one or more real-time image streams on one or more computers using one or more convolutional neural networks implemented in a Caffe framework running on one or more graphics processors.

Prior art

Currently, a large number of video recordings are made from security and other cameras. The data from them are in most cases used off-line for possible tracing of events that have been recorded in the past. Surveillance centers are used for online processing, where human power is used to analyze video sequences.

A large number of patents deal with the problem of semantic analysis (object detection) and image stream processing. For example:

• TW201605239 (A) - Video analysis method and video analysis apparatus (TOP GUIDE INTERNATE CORP) • KR20150086723 (A) - IMAGE RECODING SYSTEM (SAMSUNG TECHWIN CO LTD) • EP1955205 (Bl) - METHOD AND SYSTEM FOR PRODUCING A VIDEO SYNOPSIS (BRIEFCAM LTD) • CN104391867 (A) - Mass video searching system based on video synopsis (FUJIAN YIRONG INFORMATION TECHNOLOGY CO LTD) • US2017133053 (Al) - SYSTEMS AND METHODS FOR VIDEO SYNOPSES (FLIR SYSTEMS • US2013308921 (Al) - CREATING VIDEO SYNOPSIS FOR USE IN PLAYBACK (YAHOO INC) • KR101459103 (Bl) - METHOD FOR VIDEO MONITORING USING VIDEO SURVEILLANCE SYSTEM (DYNAMAX CO LTD)

If the above methods are used for online analysis of video recordings, in most cases one processor core is required for each security camera, resp. graphics coprocessor, which complicates the wider expansion of online analysis from a larger number of cameras.

The basic problem is how to dynamically distribute neural networks to graphics processors in order to maximize the number of processed images. Image processing means the forward passage (inference) of the neural network, when the given image is input and objects are detected at the output (ie marked location of the object, belonging to one of the specified categories).

The essence of the invention

The essence of the invention lies in the idea that it is possible to automatically decide whether to start a new network or to pair with an existing neural network on the basis of knowledge of the utilization of individual graphics processors based on the neural network request.

- 1 CZ 307563 B6

The technical solution is based on a discrete multicriteria optimization algorithm, which is implemented by dynamic programming and always finds the optimal solution, ie a scheme with the maximum balanced number of neural network instances on graphics processors, which can be achieved by a minimum number of changes in the existing layout.

The input to the scheduler algorithm is the number of graphics processors, the number of neural networks and their memory requirements. The algorithm constantly goes through a tree of all combinations of neural network layouts on individual graphics processors. The searched tree is pruned by combinations that do not meet the memory requirements. The first minimization criterion is the sum of the absolute values of the differences in the weighted numbers of neural network instances according to their configuration. The second minimization criterion used to comply with the first criterion is the number of changes required to achieve the status given by the first criterion. The best configuration found so far is kept in the scheduler's memory. The best values found in each subtree are memoized. The algorithm is invariant to layout permutations for individual graphics processors (it does not matter the order of graphics processors).

Explanation of drawings

The technical solution and its functions are described in more detail in the attached drawings:

- Giant. 1 shows a basic description of neural network assignment requirements

- Giant. 2 illustrates the role of the GPU manager in allocating computational tasks to individual graphics processors

- Giant. 3 shows the decision tree of the GPU manager

Examples of embodiments of the invention

The basic building block of the whole semantic analysis process (object detection) is the interaction of two processes, which are the proxy detector 02 and the neural network 03. At the input, the proxy detector 02 receives an image which sends to the neural network 03 via network or between process communication 04 and processes it using the interference specified in the sent configuration and again sends the detected objects back by network or between process communication 04.

The proxy-neural network detector pair is created by a neural network creation request from the proxy detector 02 sent to the GPU manager 01 process. The request 05 contains a configuration file describing the required neural network, the GPU manager deciding whether to start a new network or pair with an existing neural network. network 03.

The connection cardinality is N: 1, i.e. the proxy detector 02 can be paired with only one neural network 03, but one neural network 03 can be paired with multiple proxy detectors 02. The GPU manager 01 maintains pairing records 10, which are the communication interfaces of both processes , the process identifier and configuration of the neural network and the graphics card 09 on which the neural network performs inference.

At the input of each neural network is a dispenser 11. If more than one proxy detector is connected to the neural network, then all image processing requests are stored asynchronously in buffer 12. Each time the neural network processes all current batches of images, it selects additional collected requests from the buffer, divides them into batches (up to) a predetermined size and begin processing them.

-2EN 307563 B6

Unpairing is initiated by the proxy detector 02 by a request sent to the GPU manager 01. The GPU manager 01 cancels the pair record and in case the last pair with the given neural network disappears, then

The GPU manager 01 closes the respective neural network 03.

The scheduling of individual neural networks 03 into GPUs is performed by GPU manager 01. The task of GPU manager 01 is to ensure that all running neural networks 03 consume a similar weighted amount of computing resources provided by GPUs 09. Another condition for GPU manager 01 is to minimize the total number of changes ( creation and termination of neural networks).

The proxy detector 02 can be aggregated into assemblies 06 in one process. Detectors 02 with different and identical configurations coexist in the assemblies 06 (duplicate detectors 07). The image processing request is sent to all differently configured detectors 02 at once and the detections are unified. Conversely, of the duplicate detectors 07, only one request is sent for each image.

The duplicate detector 07 is selected for each image iteratively cyclically with a selectable distance between the duplicate detector to which the processing request is sent and the duplicate detector 07 to which the result request is sent. Suppose d the number of detectors, s the size of the distance, i the serial number of the image, then the request for image processing is received by detector i (mod d) and the request for the result by detector i-n (mod d).

How the detector 02 behaves towards the processing request or the result depends on the frameskip parameter. The frameskip parameter specifies the maximum possible number of result requests that can be ignored if the result is not already available. If the ignore result request counter reaches the frameskip value, it forces a wait for the result. The processing request is ignored if the previous request is not processed. The results from the proxy 02 detectors are sorted according to the chronological order of the respective processing requests.

If the result request is skipped due to the frameskip parameter, the missing result is replaced by extrapolating the results of the proxy 02 detectors from the previous steps. Pairing the results from individual steps and subsequent extrapolation are modeled by the Markov process based on system dynamics (laws of motion).

Figure 3 describes the scheduler decision algorithm. The input to the scheduler algorithm is the number of graphics cards, the number of neural networks and their memory requirements. The algorithm constantly goes through a tree of all combinations of neural network layouts on graphics cards. The searched tree is pruned by combinations that do not meet the memory requirements. The algorithm uses two minimization criteria: (1) the sum of the absolute values of the differences in the weighted numbers of neural network instances according to the configuration and (2) the number of changes needed to reach the state given by the first criterion. The best values found in each subtree are memoized.

Balancing (weighting) the number of neural network instances depends on the neural network inference rate on the graphics card, the number of image processing requests per time constant for a given neural network instance, and the presence of other neural network instances on the graphics card. Let s have the rate of inference, p the number of requests, then the weight coefficient can be calculated for each pair of graphical kartaneuron network

SiPi

Wi = ----, ςΑρΑ where the indices j go through all instances of neural networks on the same graphics card as i, except i itself.

-3GB 307563 B6

Industrial applicability

The presented technical solution will be used especially in the field of automated processing of recordings from security cameras in real time. Automated analysis of video images from a large number of security cameras in real time will prevent unwanted behavior or terrorist acts.

Claims (5)

PATENT CLAIMS A method of dynamically allocating computational processes to one or more graphics processors, characterized in that a scheduler, called a GPU manager, the input of which is the number of graphics processors (09), the number of neural networks (03) and their memory requirements, using a discrete multicriteria optimization algorithm always finds the optimal solution, which is a scheme with the maximum balanced number of instances of neural networks (03) on individual graphics processors (09), and which is achieved by a minimum number of changes in the existing layout of neural networks (03) on individual graphics processors (09). A method of dynamically allocating computational processes to one or more graphics processors according to claim 1, characterized in that the tree of all combinations of neural network distributions (03) on individual graphics processors is cut by combinations that do not meet memory requirements according to two minimization criteria. the sum of the absolute values of the differences in the weighted number of instances of the neural network process (03) and the number of changes necessary to reach the state given by the first criterion. Method for dynamically allocating computational processes to one or more graphics processors according to claim 2, characterized in that on the basis of a request (05) containing a configuration file describing the required neural network (03), it is decided whether a new one is started or paired existing neural network process. A method of dynamically allocating computational processes to one or more graphics processors according to claim 1, characterized in that pairing records are permanently maintained, which are communication interfaces of both processes, process identifier and neural network configuration (03) and graphics processor (09) on which the neural network (03) performs inference. A method of dynamically allocating computational processes to one or more graphics processors according to claim 2, characterized in that it comprises an automatic purification mechanism ensuring that the tree does not contain any neural network instance (03) that does not pair with running processes.