KR102011402B1 - Intelligent Face Recognition and Tracking System and Method to Distribute GPU Resource using CUDA - Google Patents

Abstract

The present invention relates to an intelligent type facial recognition and tracking system to distribute a GPU resource using CUBA, which comprises: an image processing unit configured with a GPU and performing detection and tracking of a face object in a grid manner from an image inputted through a camera; and a control unit selecting a GPU based on an index based on memory usage and the number of activated cores of each GPU through CUDA, and allocating the selected GPU to a thread. According to the present invention, the conventional face detection algorithm is replaced with a method using deep learning, and face detection and recognition is possible in real time through efficient resource distribution by eliminating the increasing amount of operation with the GPU resource distribution. Accordingly, the waste of human resources and the time controlling a CCTV or the like may be reduced, and services such as identification and tracking of an airport blacklist, early resolution of disappearance accidents, and the like are possible.

Description

Intelligent Face Recognition and Tracking System and Method to Distribute GPU Resource using CUDA}

The present invention relates to an intelligent face recognition / tracking system and method for distributing graphics processing unit (GPU) resources using Compute Unified Device Architecture (CUDA), and more particularly, to a conventional face recognition and tracking system. To overcome the shortcomings and to configure the multiple GPUs to improve the stability and performance of the system, and to efficiently distribute the GPU resources to find the target to find in the real-time video, the technology that can be tracked.

First, the face recognition and tracking system uses four algorithms for face detection, the second is color-based tracking, the third is face recognition in sampled images, and the fourth is no allocation of GPU resources. However, the conventional face recognition and tracking system has the following problems.

First, the method of using the face detection algorithm has a problem in that the face is detected by a different algorithm according to the angle of the face because the characteristics vary according to the angle of the face.

In addition, the color-based tracking method utilizes the color information of the current RoI region, and there is a problem that the tracking failure probability is very high when the tracking target appears covered by an obstacle such as a pillar or goes out of the camera field of view. .

In addition, the face recognition method in the sampled image, there is a problem that the detection and recognition rate is significantly reduced when the image is shaken or out of focus in the sampled image is not real-time recognition.

In addition, in a method of not distributing GPU resources, a problem does not occur at a time when the amount of computation is small, but a problem occurs in the stability of the system due to a slow speed or an overload of the GPU when a large amount of computation occurs.

Korea Patent Registration No. 10-1850286

An object of the present invention is to replace an algorithm for face detection used in a conventional face recognition and tracking system with a deep learning method, and to solve the problem by using a GPU resource distribution algorithm that efficiently distributes an increasing amount of computation. The distribution enables face detection and recognition in real time, and accordingly, the detection and recognition is repeated to implement a real time tracking system.

An object of the present invention is to provide a real-time tracking system by applying the GPU resource distribution algorithm, to reduce the waste of manpower, time waste to control CCTV, etc., to enable services such as airport blacklist identification and tracking, early resolution of missing accidents It is.

An intelligent face recognition and tracking system for distributing GPU resources using a CUDA according to an embodiment of the present invention for achieving the technical problem, is configured with a GPU to detect the face object in a grid method from the image input through the camera And an image processor which performs tracking. And a controller configured to select an optimal GPU based on an index based on memory usage and the number of activated cores of each GPU through CUDA, and allocate the selected GPU to a thread.

Preferably, the image processor detects the object from the image input through the camera in a grid manner, and determines whether the detected object matches the target to be found, but if the detected face object matches the target to be found, The RoI (Region of Interest) may be designated in the form of a box in the matching image.

The image processor may perform tracking on the RoI by repeatedly performing a procedure of detecting a face object from the input image and determining whether the detected face object matches the target to meet a predetermined criterion.

The image processor may adjust the angle of view of the camera such that the RoI is positioned at the center of the angle of view through PTZ control when the image is out of a predetermined range from the center point of the camera angle of view during tracking for the RoI.

The controller derives the current usage of each GPU based on the memory usage of each GPU and the number of activated cores, and allocates the GPU with the lowest share to the thread, but the share of the GPU to which the thread is allocated becomes higher than the preset level. In addition, the current usage of each GPU is derived to reallocate the GPU with the lowest share to the thread.

An intelligent face recognition and tracking method for distributing GPU resources using a CUDA according to an embodiment of the present invention based on the above-described system includes: (a) distributing a plurality of GPU resources by a controller; And (b) outputting, by the display unit, an image received by the image processor, a face object detected from the image, and tracking.

Preferably, the step (a) includes the step (a-1) of determining, by the control unit, how much each GPU is currently used by detecting the memory usage of each GPU and the number of activated cores; (A-2) by the controller, allocating a GPU having the lowest occupancy rate to a thread according to a current resource usage state of each GPU; (A-3) determining, by the controller, whether the GPU occupancy is higher than a predetermined level; And (a-4) when the controller occupies the GPU occupancy level higher than or equal to a predetermined level as a result of step (a-3).

Step (b-1) may include detecting a face object in a grid manner from an input image by the image processor; (B-2) determining whether the face object detected by the image processor matches the target to be searched for; (b-3) designating a region of interest (RoI) in the form of a red box on the matched image when the detected face object matches the target to be found as a result of the determination of step (b-2); (B-4) performing the tracking on the RoI designated by the image processor; (B-5) determining whether the image processor deviates by a predetermined range or more from a center point of the camera angle of view during tracking for a specified RoI; As a result of determining in step (b-6), if the camera is out of a predetermined range from the center point of view, the image processing unit adjusts the camera angle of view so that the corresponding RoI is located at the center of the field of view through PTZ (Pan Tilt Zoom) control. It is characterized by including.

According to the present invention as described above, by replacing the face detection algorithm used in the conventional face recognition and tracking system with the method using deep learning, by solving the GPU resource distribution algorithm that effectively distributes the increased amount of computation, Efficient resource distribution enables face detection and recognition in real time, and thus it is possible to implement a real-time tracking system by repeating detection and recognition.

According to the present invention, by applying the GPU resource distribution algorithm to provide a real-time tracking system, it is possible to reduce the labor waste, time waste, CCTV blacklist identification and tracking, early resolution of missing accidents, such as controlling CCTV have.

1 is a block diagram illustrating an intelligent face recognition and tracking system for distributing GPU resources using CUDA in accordance with one embodiment of the present invention.
2 is a flowchart of a GPU resource distribution algorithm of an intelligent face recognition and tracking system for distributing GPU resources using CUDA in accordance with one embodiment of the present invention.
3 is an exemplary diagram illustrating camera angle adjustment of an intelligent face recognition and tracking system for distributing GPU resources using a CUDA according to an embodiment of the present invention.
4 is a flow diagram of an intelligent face recognition and tracking system for distributing GPU resources using CUDA in accordance with one embodiment of the present invention.
5 is an exemplary diagram of GPU resource usage of an intelligent face recognition and tracking system for distributing GPU resources using CUDA in accordance with one embodiment of the present invention.
6 is an exemplary diagram illustrating optimal GPU allocation to thread # 1 of an intelligent face recognition and tracking system using CUDA to distribute GPU resources, in accordance with an embodiment of the present invention.
7 is a flowchart of optimal GPU allocation for all threads of an intelligent face recognition and tracking system using CUDA to distribute GPU resources, in accordance with an embodiment of the present invention.
8 is an exemplary diagram illustrating a process of reallocating GPU # 2 to thread # 1 of an intelligent face recognition and tracking system using CUDA according to an embodiment of the present invention.
9 is a diagram illustrating an internal network of an intelligent face recognition and tracking system for distributing GPU resources using a CUDA according to an embodiment of the present invention.
10 is an exemplary diagram illustrating a face region detection process using deep learning of a tracking system and intelligent face recognition for distributing GPU resources using a CUDA according to an embodiment of the present invention.
FIG. 11 is an exemplary diagram illustrating a face recognition process using intelligent face recognition and deep learning of a tracking system for distributing GPU resources using a CUDA according to an embodiment of the present invention. FIG.
12 is a network explanatory diagram used in a learning process of an intelligent face recognition and tracking system for distributing GPU resources using a CUDA according to an embodiment of the present invention.
FIG. 13 is an explanatory diagram of a network used in an intelligent face recognition and tracking system for distributing GPU resources using a CUDA according to an embodiment of the present invention. FIG.
FIG. 14 is an exemplary diagram illustrating real-time face tracking result image using deep learning of a tracking system and intelligent face recognition for distributing GPU resources using CUDA according to an embodiment of the present invention. FIG.
15 is an exemplary view comparing an angle of view of a general camera and a PTZ camera of an intelligent face recognition and tracking system using GPU to distribute GPU resources according to an embodiment of the present invention.
FIG. 16 is a diagram illustrating tracking target tracking through PTZ control of a tracking system and intelligent face recognition using GPU to distribute GPU resources according to an embodiment of the present invention. FIG.
FIG. 17 is a result graph according to the GPU resource distribution difference of the intelligent face recognition and tracking system for distributing GPU resources using CUDA according to an embodiment of the present invention. FIG.
18 is a graph showing GPU occupancy check results before and after GPU resource allocation in a tracking system and intelligent face recognition using GPU to distribute GPU resources according to an embodiment of the present invention.
19 is a flow chart illustrating an intelligent face recognition and tracking method for distributing GPU resources using CUDA in accordance with one embodiment of the present invention.
20 is a flowchart illustrating step S100 of an intelligent face recognition and tracking method for distributing GPU resources using a CUDA according to an embodiment of the present invention.
21 is a flowchart illustrating step S200 of an intelligent face recognition and tracking method for distributing GPU resources using a CUDA according to an embodiment of the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, the terms or words used in the present specification and claims are defined in the technical spirit of the present invention on the basis of the principle that the inventor can appropriately define the concept of the term in order to explain his invention in the best way. It should be interpreted to mean meanings and concepts. In addition, when it is determined that the detailed description of the known function and its configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description is omitted.

As shown in FIG. 1, an intelligent face recognition and tracking system S for distributing GPU resources using a CUDA according to an embodiment of the present invention is composed of a plurality of GPUs and inputted through a camera 1. Through the image processing unit 100 and CUDA, which detects and tracks face objects in a grid method from images, the optimal GPU is selected and selected based on the memory usage and the number of activated cores of each GPU. The controller 200 may be configured to allocate the GPU to a thread, and the display 300 may output an image received by the image processor 100, a face object detected from the image, and tracking.

In this case, a series of processes for an algorithm for optimal GPU resource distribution among a plurality of GPUs using CUDA is illustrated in FIG. 2.

In addition, CUDA is used as an API (Application Programming Interface) for using the GPU, and the reason for using CUDA is that the GPU is suitable for processing a large amount of associations at the same time. Compared to just one core, the CPU's computational speed is superior, but the commonly used CPU is 4 to 8 cores, whereas the GPU has more than 1500 cores.

Therefore, a large amount of operations can be simultaneously processed using the GPU, and in one embodiment of the present invention, a CUDA is applied as an API that makes the GPU easy to program, and the image processor 100 includes at least three GPUs. It is configured to.

Referring to FIG. 3, the image processing unit 100 is configured of a multi-GPU including a plurality of GPUs to perform a task according to the GPU resource distribution of the control unit 200. However, the image processing unit 100 performs a grid method from an image input through the camera 1. The facial object is detected, and it is determined whether the detected facial object matches the target to be found.

In this case, the object detected by the image processor 100 may be set to either a face region or a specific object, and the determination of whether the detected object matches the target to be searched is derived through deep learning.

If the detected face object matches the target to be searched for, the image processor 100 designates a region of interest (RoI) in the form of a red box on the matched image, and displays name information of the face object on the box. At this time, the color of the RoI designation box may be changed by user setting.

The image processor 100 conforms to a predetermined criterion for assigning a thread to the optimal GPU described above, detecting a face object from an image input through the camera 1, and determining whether the detected face object matches the target. Repeatedly executes tracking for the specified RoI.

If the image processing unit 100 deviates from the center point of the camera 1 angle of view more than a predetermined range during tracking for the designated RoI, the RoI is moved to the center of the field of view through PTZ (Pan Tilt Zoom) control as shown in FIG. 4. Adjust the angle of view of the camera 1 to position it.

In this case, the camera 1 may be configured in a number corresponding to the GPU provided in the image processing unit 100.

Hereinafter, the controller 200 looks at the detailed procedure for controlling the GPU device of the image processor 100 as follows.

First, as shown in FIG. 5, the controller 200 detects how much each GPU is currently used by detecting memory usage and number of activated cores of each GPU.

Secondly, the controller 200 allocates a GPU having the lowest occupancy rate to a thread according to the current resource usage state of each GPU.

At this time, the optimal GPU is allocated in consideration of both the memory usage and the number of activated cores, and as shown in FIG. 6, the control unit 200 allocates GPU # 2 having the lowest resource occupancy to thread # 1. do.

In addition, the controller 200 performs the first and second processes on the threads # 2 and the threads # 3 to allocate the optimal GPU to all the threads as shown in FIG.

In this case, after allocating an optimal GPU to all threads, the controller 200 may cause an overload of the GPU when the first and second processes are repeated every frame.

Therefore, thirdly, the controller 200 performs the first and second processes again only when the GPU occupancy becomes higher than a predetermined level.

In this case, when the algorithm is executed by increasing the reference value, the GPU occupancy may increase faster than expected, and there may be an overload before switching the GPU. On the contrary, when the reference value is lowered, GPU switching occurs too frequently, causing performance degradation.

Therefore, the criteria for performing the above-described first and second processes again may be assumed to be 80% of the GPU occupancy, and this criterion may be changed according to the performance of the GPU although it is derived by experiment.

As illustrated in FIG. 8, the controller 200 allocates GPU # 2 having the lowest resource occupancy to thread # 1, but the resource occupancy ratio is over 80% of the GPU occupancy, which is the criterion for performing GPU # 2 again. Configured to reassign the lowest GPU # 1 to thread # 1.

9 to 13, an image processing unit according to an embodiment of the present invention will be described below with reference to a procedure of detecting and recognizing a face region using deep learning.

The Haar-like feature algorithm, widely used as a conventional face detection algorithm, detects facial areas by using characteristic brightness differences between areas such as eyebrows and pupils, and is basically used to maintain the geometric information of an object. Or a slight change in position, indicating strength. However, it has a weak point in a situation in which the brightness of the image changes drastically, when the target area is rotated, or when the color contrast of each area of the face is not clear.

Therefore, in an embodiment of the present invention, to overcome this disadvantage, the face region is trained by deep learning up to various angles of rotation.

Learning information for detecting the position of the face region is a total of five channels. The first is a channel that recognizes which object, the second is the x value of the center coordinates in the location information, the third is the y value of the center coordinates in the location information, the fourth is the width w of the object as a ratio, and the fifth is the The height h value.

That is, one embodiment of the present invention is the same as the existing deep learning process, but there is a difference in the way of detecting the object. Object detection is based on grid detection, unlike deep learning with neural networks.

One embodiment of the present invention has an advantage that the processing speed is high because the input image is divided into grids.

As shown in FIG. 9, the deep learning network used for face region detection is composed of 26 layers, and the learned data are used for face detection in a real-time image.

The image processor displays a portion determined to be a face in the image input in real time from the camera in the form of a white box, and cuts it out temporarily for the next step, face recognition.

The face region detection of the image processor is divided into two types and object detection is performed. First, the total size of the input image is calculated as '1'. Accordingly, the center coordinate value, width, and height are derived from the decimal point.

At this time, the input image is stored in five types of x, y, w, h and class, which are generated as many as the number of learned classes, and the final object is detected with the recognition rate in the generated data.

That is, as in the example of the object detection process illustrated in FIG. 10, the entire flow of face region detection using deep learning and 20 classes may be included.

In addition, face recognition using deep learning, as shown in FIG. 11, the image processing unit detects a face region of a target to be found in a photograph or an image using a face detection algorithm described above, classifies classes, and stores and learns them. do.

Subsequently, after the image processing unit finishes the learning, the image processing unit receives the image of the temporarily stored face region from the image input in real time and recognizes whether the object matches the class to be searched for.

When the image processor finds the third object to be searched for, the image processor displays the image in the form of a red box and displays the name of the object on the box.

In an embodiment of the present invention, the convolution filter and the fully continued layer are fine-tuned to improve the recognition result. To prevent overfitting, the Maxpooling layer is added, and the train set and the test set are divided and learned.

The train set modifies the weights using the backpropagation algorithm during the learning process, and the test set determines whether the weights are overfitting using the DB different from the train set when learning a certain number of times. Train sets and test sets are trained using Softmax on the last layer.

Finally, after the learning is completed, the feature vector is extracted through the layer only in the forward direction without the backpropagation algorithm, and the recognition is performed using the feature vector.

In this case, the size and role of each layer of the network used for face recognition are as shown in FIG. 12, and as shown in FIG. 13, the deep learning network used for face recognition consists of 16 layers.

Hereinafter, a procedure for tracking a face according to an exemplary embodiment of the present invention will be described with reference to FIG. 14.

Existing template matching has a disadvantage that the matching fails when the shape of the object is changed and the face angle is changed.In the HOG algorithm, the RoI region of the entire frame is extracted and used. As a result, the RoI area is vulnerable to being obscured by an obscured image.

Therefore, in an embodiment of the present invention, the above-described processes (GPU allocation, face detection, and face recognition) are repeated to effectively perform real-time face tracking.

Accordingly, if the image processing unit repeats the GPU allocation, face detection, and face recognition process, even when the RoI region disappears or appears inside the camera view angle, tracking is efficiently performed. As shown in FIG. As a result of real-time face tracking using running, when an object to be searched is found, the image is displayed in a red box form, otherwise it is displayed in a white box form.

Hereinafter, the camera control according to an embodiment of the present invention will be described with reference to FIGS. 15 and 16 as follows.

As shown in FIG. 15, the camera according to an embodiment of the present invention is configured to enable Pan Tilt Zoom (PTZ) control to have a wider angle of view than a conventional fixed camera, and to intensively track an object to be searched for. Can be. Thus, even with a single camera configuration, it is possible to recognize and track in a range that multiple cameras should watch.

In addition, as shown in FIG. 16, when the tracking target deviates by a predetermined distance or more from the center of the angle of view, the tracking target may be controlled to be positioned at the center of the angle of view again through PTZ control.

Hereinafter, with reference to FIGS. 17 and 18, an intelligent face recognition and tracking system (S) for distributing GPU resources using CUDA according to an embodiment of the present invention for GPU resource distribution, face detection, face recognition, and tracking The results of the experiment are as follows.

First, to measure the effectiveness of intelligent face recognition and tracking system that distributes resources using CUDA, we built a database of about 150 people with different genders, ages, and body types and trained them using deep learning.

After the training is completed, the GPU resource is divided into and when it is not distributed, and in each case, the number of people appearing in the input image is changed from 1 to 5, displaying the face detection rate, recognition rate, and recognition result. The fps of the video was measured and the degree of activation of the GPU was checked.

Original camera video was input at 30fps. As shown in [Table 1], when the GPU resources are not distributed, the detection and recognition rate decreases rapidly as the number of people appearing in the image increases.If more than four people appear, the image does not load at all. Calculated.

In Table 2, the experimental results when the GPU resources were distributed were constant regardless of the number of faces in the image and the recognition rate, and the fps of the resultant video was also maintained at about 25 fps.

The result according to the GPU resource distribution difference shows a more obvious difference in the graph of FIG.

Table 3 shows the GPU occupancy before GPU resource distribution, and Table 4 shows the GPU occupancy after GPU resource distribution. 18 is a graph illustrating GPU occupancy states according to GPU resource distribution differences.

When the GPU resources were not distributed, the situation in which the video was not output as the usage of the first GPU went up to 100% in an instant, caused no further experiments. You can also see that GPUs 2 and 3 weren't even loaded at all during the experiment.

In the case of distributing GPU resources, it can be seen that the GPU resources 1, 2, and 3 are evenly used. As the number of appearances increases, the amount of computation increases, and as the amount of computation increases, the difference in occupancy and experimental results widens depending on whether the GPU resources are distributed. Therefore, the effectiveness of intelligent face recognition and tracking system for distributing resources using the proposed CUDA has been proved.

In addition, as shown in Table 5, in the stationary state, about 98.2% in the front, about 97.3% in the side, about 94.6% in the front and 91.7% in the side at the slow pace (1.0m / sec). , Average of about 93.4% for the front side, about 89.1% for the side, and about 93.1% for the front side, and about 89% for the side at normal walking speed (1.3m / sec) The recognition rate was confirmed. The sides were measured within the range of 1 kPa to 45 kPa rotation.

Hereinafter, an intelligent face recognition and tracking method for distributing GPU resources using a CUDA according to an embodiment of the present invention will be described with reference to FIG. 19.

First, the controller 200 distributes a plurality of GPU resources (S100).

The display unit 300 outputs an image received by the image processing unit 100, a face object detected from the image, and tracking (S200).

Hereinafter, referring to FIG. 20, a detailed process of step S100 of the intelligent face recognition and tracking method for distributing GPU resources using a CUDA according to an embodiment of the present invention will be described below.

First, the controller 200 detects the memory usage of each GPU and the number of activated cores to determine how much each GPU is currently used (S102).

Subsequently, the controller 200 allocates the GPU having the lowest occupancy rate to the thread according to the current resource usage state of each GPU (S104).

Subsequently, the controller 200 determines whether the GPU occupancy is higher than a predetermined level (S106).

As a result of the determination in step S106, when the GPU occupancy becomes higher than the predetermined level, the controller 200 performs the procedure to the step S102 (S108).

Hereinafter, referring to FIG. 21, a detailed process of step S200 of the intelligent face recognition and tracking method for distributing GPU resources using a CUDA according to an embodiment of the present invention will be described below.

After operation S100, the image processor 100 detects a face object from the input image in a grid manner (S202).

Next, the image processor 100 determines whether the detected face object matches the target to be searched for (S204).

As a result of the determination in step S204, when the detected face object matches the target to be searched, the image processor 100 designates a region of interest (RoI) in the form of a red box on the matched image (S206). In this case, name information of the face object may be displayed on the box.

Subsequently, the image processing unit 100 performs tracking on the designated RoI (S208).

Subsequently, the image processor 100 determines whether the camera 1 is out of a predetermined range or more by a predetermined range during tracking for the designated RoI (S210).

As a result of the determination in step S210, when the camera 1 is out of a predetermined range from the center point of view, the image processing unit 100 controls the corresponding RoI to the center of the field of view through PTZ (Pan Tilt Zoom) control. Adjust the angle of view (S212).

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many modifications and variations can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes, modifications, and equivalents should be considered to be within the scope of the present invention.

S: Intelligent face recognition and tracking system using CUDA to distribute GPU resources
1: camera
100: image processing unit
200: control unit
300: display unit

Claims (8)

An image processor configured to include a GPU and detect and track a face object in a grid manner from an image input through a camera; And
A control unit for selecting a GPU based on an indicator of memory usage and the number of activated cores of each of the GPUs through a CUDA, and allocating the selected GPUs to threads;
The image processor,
Detecting an object in a grid method from the image input through the camera,
Determining whether the face object detected through deep learning matches the target to be searched for,
Determine whether the detected object matches the target you are looking for,
If the detected face object matches the target to be searched for, the RoI (Region of Interest) is assigned to the matched image in the form of a box.
The control unit,
Based on the memory usage of each GPU and the number of active cores, the current usage of each GPU is derived and the GPU having the lowest resource occupancy is allocated to threads.
Each time the face object is detected and tracked, the current usage of each GPU is derived, and when the share of the GPU to which the thread is allocated becomes higher than a predetermined level, the current usage of each GPU is derived to determine the GPU having the lowest resource share. Intelligent face recognition and tracking system that uses CUDA to distribute GPU resources, which is reassigned to threads. delete The method of claim 1,
The image processor,
It is characterized in that the tracking of the RoI is performed by repeatedly performing a procedure of detecting a face object from the input image and determining whether the detected face object matches a target to meet a predetermined criterion.
Intelligent face recognition and tracking system that uses CUDA to distribute GPU resources. The method of claim 1,
The image processor,
If the deviation from the center point of the camera angle of view more than a predetermined range during tracking for the RoI, the angle of view of the camera is adjusted so that the RoI is located in the center of the field of view through PTZ control
Intelligent face recognition and tracking system that uses CUDA to distribute GPU resources. delete delete delete delete

Images