KR101534776B1 - A Template-Matching-Based High-Speed Face Tracking Method Using Depth Information - Google Patents

Abstract

A template matching-based high-speed image processing unit that receives a depth image and a color image having frames consecutive in time from a camera that photographs a face of a person, detects the face in one frame to generate a template, A method for tracking a face, the method comprising: (a) comparing a depth of the template with a depth of a current frame to update a size of the template; (b) setting a region of the current frame extending to a predetermined ratio (hereinafter referred to as expansion ratio) of the size of the updated template as a search region, with the updated template as a center; (c) performing template matching in the search area with the updated template and selecting a matching position; And (d) repeating the steps (a) to (c) by updating the matched position to the position of the updated template and using the next frame of the current frame as a current frame. .
According to the template matching based high speed face tracking method as described above, by applying early termination and sparse search and reducing the search area by adjusting the template size, the speed of face tracking is increased to process in real time, To improve the accuracy of tracking.

Description

{Template-Matching-Based High-Speed Face Tracking Method Using Depth Information}

The present invention relates to a method for tracking a face at high speed using only depth information, which uses a template matching method and applies an early termination method and a sparse search method, And more particularly, to a template matching based high speed face tracking method for performing matching correction.

In particular, the present invention estimates a depth value of a face to be tracked in order to correct a change in depth according to a movement of a face, adjusts the size of the template according to a result of the estimation, And more particularly, to a template-matching based high-speed face tracking method for adjusting regions.

Methods of tracking a part of a human body have been studied for a long time in various fields including computer vision field and widely used for security systems, video conferencing, robot vision, interactive system by HCI (human-computer interface) [Non-Patent Document 1] [Non-Patent Document 2]. Among these, research on the face has been actively studied, and its purpose is fast and accurate tracking.

Face tracking detects moving faces of a moving person in a sequence of moving images and traces the movement path, and focuses on fast running speed in real - time environment.

The simplest way to track a face is to look at the face as an object and match the block corresponding to the object [Non-Patent Document 3]. In addition, there is a method of tracking moving objects in a dynamic background by using a preprocessing or modeling method using mathematical and physical phenomenon to track the most similar objects in an image [Non-Patent Document 4-6]. [Non-Patent Document 4] uses combined information of color and motion for strong facial analysis in a feature-based method, and Non-Patent Document 5 describes a method for tracking facial pose with a video sequence, The framework was combined. [Non-Patent Document 6] proposed a stable and robust technique for 3D head tracking under various illumination conditions.

Unlike the conventional method using two-dimensional images, a method of using three-dimensional motion [Non-Patent Document 7] has been studied to use three-dimensional information.

[Non-Patent Document 7] proposed a method of tracking a person with 3-D transformation or rotation information by integrating optical flow and depth. Non-Patent Document 8 [Non-Patent Document 9] [Non-Patent Document 10] methods of using mutation values by stereo matching were also studied in order to use three-dimensional information as well as three-dimensional motion.

[Non-Patent Document 8] proposes a 3D face tracking based on a stereo vision camera system that does not require a preprocessing (calibration) process for 3D image acquisition through a stereo camera. [Non-Patent Document 9] proposed a method of combining the advantages of color distribution and depth. [Non-Patent Document 10] proposed a PCA-based face recognition algorithm using displacement information extracted from a stereo camera.

Lighting changes in tracking objects are very sensitive and are a problem in tracking. However, by using the 3D information, that is, the depth information, it is possible to solve the problem caused by the illumination change due to strong resistance to various illumination changes.

In recent years, SR4000 [Non-Patent Document 11] [Non-Patent Document 12] of TOF (Time of filght) method or Kinect [Non-Patent Document 12] of structured optical system of Microsoft Since the depth information can be obtained in real time using the document [13], studies are being conducted to directly use the depth information obtained from the depth information on the face tracking.

[Non-Patent Document 14] proposed a method of tracking a person's nose using the feature-based method SR3000. [Non-Patent Document 15] uses a template matching method for tracking a face using the depth of a template using Kinect and SR4000. Also, a method of reducing the complexity of motion estimation using a depth camera and a method of accurately estimating zoom motion using distance information between a current block and a reference block are proposed [Non-Patent Document 19].

Kinect-based methods also require a more accurate and faster method for real-time face detection and tracking. In particular, for this purpose, it is necessary to implement an early termination method or a method capable of applying sparse search methods well.

G, Q, Zhao, et al., "A Simple 3D Face Tracking Method Based on Depth Information," Int'l Conf. on Machine Learning and Cybernetics, pp. 5022-5027, Aug. 2005. [Non-Patent Document 2] C. X. Wang and Z. Y. Li, "A New Face Tracking Algorithm Based on Local Binary Pattern and Skin Color Information," ISCSCT, Vol. 2, pp. 20-22, Dec. 2008. [Non-Patent Document 3] K. Hariharakrishnan and D. Schonfeld, "Fast object tracking using adaptive block matching," IEEE Trans. Multimedia, vol. 7, no. 5, 2005. [Non-Patent Document 4] M. Lievin and F. Luthon; "Nonlinear Color Space and Spatiotemporal MRF for Hierarchical Segmentation of Face Features in Video," IEEE Trans. Image Processing, vol. 13, No. 1, Jan. 2004. [Non-Patent Document 5] Y. Lin et al., "Real-time Face Tracking and Pose Estimation with Partitioned Sampling and Relevance Vector Machine," IEEE Intl. Conf. Robotics and Automation, pp. 453-458, 2009. [Non-Patent Document 6] A. An and M. Chung, " Robust Real-time 3D Head Tracking Based On Online Illumination Modeling and its Application to Face Recognition, IEEE Intl. Conf. Intelligent Robots and Systems, pp. 1466-1471, 2009. [Non-Patent Document 7] R. Okada, Y. Shirai, and J. Miura, "Tracking a Person with 3-D Motion by Integrating Optical Flow and Depth," Proc. Fourth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 336-341,2000. [Non-Patent Document 8] G. Zhao, et al., "A Simple 3D Face Tracking Method Based on Depth Information," Intl Conf. Machine Learning and Cybernetics, pp. 5022-5027, 2005. [Non-Patent Document 9] Y. H. Lee et al., "A Robust Face Tracking using Stereo Camera," SICE Annual Conf., Pp. 1985-1989, Sept. 2007. [Non-Patent Document 10] S. Kosov et al., "Rapid Stereo-vision Enhanced Face Recognition," IEEE Intl. Conf. Image Processing, pp. 2437-2440, Sept. 2010. [Non-Patent Document 11] Mesa Imaging, SR4000 user manual v2.0, May 2011. [Non-Patent Document 12] M. Hacker, et al., "Geometric Invariants for Facial Feature Tracking with 3D TOF Cameras," Int'l Symposium on Signals, Circuits and Systems, Vol. 1, pp. 1-4, 2007. [Non-Patent Document 13] J. L. Wilson, Microsoft Kinect for Xbox 360, PC Mag. Com, Nov. 10, 2010. [Non-Patent Document 14] M. Hacker, et al., "Geometric Invariants for Facial Feature Tracking with 3D TOF Cameras," Int'l Symposium on Signals, Circuits and Systems, Vol. 1, pp. 1-4, 2007. [Non-Patent Document 15] X. Suau, J. Ruiz-Hidalgo and J. Casas, "Real-Time Head and Hand Tracking Based on 2.5D Data", IEEE Trans. Multimedia, Vol. 14, No. 3, pp. 575-585, June 2012. [Non-Patent Document 16] Y.-J. Bae, H.-J. Choi, Y.-H Seo and D.-W. Kim, "A Fast and Accurate Face Detection and Tracking Method by using Depth Information," J. Korean Institute of Communications and Information Science (KICS), Vol. 37A, No. 07, pp. 586-599 [Non-Patent Document 17] P. Viola and M. J. Jones, "Robust Real-Time Face Detection," Computer Vision, Vol. 52, No. 2, pp. 137-154, 2004. [Non-patent Document 18] S.-K. Kwon and S.-W. Kim, "Motion Estimation Method by Using Depth Camera," J. Broadcast Engineering, Vol. 17, No. 4, pp. 676-683, Jul. 2012. [Non-Patent Document 19] S.-K. Kwon, Y.-H. Park and K.-R. Kwon, "Zoom Motion Estimation Method by Using Depth Information," J. Korea Multimedia Society, Vol. 16, No. 2, pp. 131-137, Feb. 2013.

SUMMARY OF THE INVENTION The object of the present invention is to solve the above-mentioned problems, and an object of the present invention is to provide a method of tracking a face at high speed using only depth information by using a template matching method and applying an early termination method and a sparse search method And a matching method based on template matching to perform matching correction on neighboring pixels in order to correct a tracking error.

It is another object of the present invention to estimate a depth value of a face to be tracked in order to correct a change in depth according to movement of a face, to adjust the size of the template according to a result of the estimation and to perform template matching according to the size of the adjusted template Speed face tracking method based on template matching that adjusts the search area to be scanned.

According to an aspect of the present invention, there is provided a depth-of-field image processing method including receiving a depth image and a color image having frames successive in time from a camera that photographs a face of a person, detecting the face in one frame to generate a template, (A) comparing a depth of the template with a depth of a current frame to update a size of the template; (b) setting a region of the current frame extending to a predetermined ratio (hereinafter referred to as expansion ratio) of the size of the updated template as a search region, with the updated template as a center; (c) performing template matching in the search area with the updated template and selecting a matching position; And (d) repeating the steps (a) to (c) by updating the matched position to the position of the updated template and using the next frame of the current frame as a current frame .

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein in the step (a), each of the template and the region corresponding to the template in the current frame , Subblocks having the greatest depth are obtained with respective depths, and the size of the template is adjusted in proportion to the difference in depth between the template and the sampling region.

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein in step (c), the updated template and a sum-of-absolute differences (SAD) And selects the position having the minimum value as the matched position.

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein in step (c), when the updated template and the SAD (PSAD) per pixel in the search area are smaller than a predetermined threshold, .

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein in step (c), the searching is performed in a spiral direction gradually starting from a position of a search area corresponding to a position of the template.

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein in step (c), template matching with the updated template is performed while sequentially moving from the search area to a neighboring location, The present invention is characterized in that, when moving from a position of a pixel to a position of a pixel to be searched next, the next pixel is selected and searched beyond a pixel at a predetermined interval.

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein, in the step (c), when a pixel at a matching position is selected, further matching is performed with respect to a pixel adjacent to the selected position .

According to another aspect of the present invention, there is provided a high-speed face tracking method based on template matching, wherein in the step (c), upper and lower or upper, lower, left, and right pixels of a selected position are corrected.

The present invention also relates to a computer-readable recording medium on which a program for performing a fast matching method based on template matching is recorded.

As described above, according to the template matching-based high-speed face tracking method according to the present invention, by applying early termination and sparse search and reducing the search area by adjusting the template size, At the same time, matching correction and the like are performed to improve the tracking accuracy.

As a result of the experiments using the self-produced test sequences and the multi-point test sequences provided by MPEG, the self-produced (640 × 480) Sequence showed about 3% tracking error and 2.45 ms execution time. In Lovebird1 (1024 × 768) sequence, tracking error was about 1% and execution time was 7.46 ms.

1 is a diagram showing a configuration of an overall system for carrying out the present invention.
FIG. 2 is a flowchart for explaining a relationship between an operation of face detection and face tracking according to an embodiment of the present invention; FIG.
3 is a flowchart illustrating a template matching-based high-speed face tracking method according to an embodiment of the present invention.
4 is a pseudo code representing a face tracking method according to the present invention.
5 is a graph showing the relationship between object size and depth according to the present invention.
6 is an exemplary diagram for setting a search area according to the present invention;
Figure 7 is an illustration of a spiral sparse search method in accordance with the present invention.
8 is an exemplary view of a method of performing matching correction according to the present invention.
9 is a table for a test sequence used for parameter determination according to the present invention;
10 shows, as each test sequence image according to the present invention, LR; (a) 64th, (b) 79th, (c) 186th, UD; (d) 38th (e) 96th (f) 107th, BF; (g) 32th (h) 65th (i) Image of 149th
11 is a graph illustrating template rebalance errors for template partitioning in accordance with the present invention;
12 is a graph showing the tracking error (%) for the search range? (%) According to the present invention.
13 is a graph showing experimental results for determining an early termination threshold according to the present invention.
14 is a table showing the tracking error and the execution time for a sparse search interval according to the present invention;
15 is a table showing sequence information used for tracking according to the experiment of the present invention.
16 is a table showing experimental results of the face tracking method of the present invention in accordance with the experiment of the present invention.
17 is a resultant image traced according to the experiment of the present invention, in which WL; (a) 38th, (b) 186th, (c) 225th, S & (d) 51th, (e) 188th, (f) 292th, Lovebird1; (g) 26th, (h) 96th, (i) 135th.
18 is a table showing the performance comparison between the present invention and the conventional method according to the experiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings.

In the description of the present invention, the same parts are denoted by the same reference numerals, and repetitive description thereof will be omitted.

First, examples of the configuration of the entire system for carrying out the present invention will be described with reference to Fig.

1, the template matching based high speed face tracking method according to the present invention includes a depth image 61 photographed by the depth camera 21 of the keynote 20, (Or an RGB camera) 22 to receive a color image 62 and track the face. That is, the face tracking method may be implemented by a program and installed in the computer terminal 30 and executed. The program installed in the computer terminal 30 can operate as one program system 40. [

Meanwhile, as another embodiment, the face tracking method may be implemented by a single electronic circuit such as an ASIC (on-demand semiconductor) in addition to being operated by a general-purpose computer. Or a dedicated computer terminal 30 dedicated to only detecting and tracking the face image in the image. This is called a face tracking device 40. Other possible forms may also be practiced.

The keynote 20 includes a depth camera 21 and a color camera 22.

The depth camera 21 is a camera for measuring the depth of the object 10, and measures the depth information to output a depth image.

Preferably, the depth camera 21 is a depth camera installed in the Kinect and is a depth camera that measures depth information by an infrared pattern. The depth camera 21 is composed of an infrared ray transmitting unit and a receiving unit. When the infrared ray emitted from the transmitting unit is reflected by the object 10, the depth camera 21 receives the infrared rays reflected by the receiving unit and measures the depth of the object 10.

The photographed depth image 61 is a depth image photographed by the depth camera 21.

The color camera 22 is a conventional RGB camera and acquires the color of the object 10. [ Preferably, the color camera 22 is an RGB camera installed in the Kinect. The photographed color image 62 is an RGB image photographed by the color camera 22.

The object to be photographed by the cameras 21 and 22 is a person's face. That is, the cameras 21 and 22 are installed on the front of a person to photograph a face of a person. For example, the cameras 21 and 22 may be installed so as to face the front of the video equipment in order to detect the face of the viewer who views the 3D video equipment and to process the 3D image according to the viewpoint of the viewer.

The depth image 61 and the color image 62 are directly input to and stored in the computer terminal 30 and processed by the face tracking device 40. Alternatively, the depth image 61 and the color image 62 may be stored in advance in the storage medium of the computer terminal 30 and read from the depth image 60 stored by the face tracking device 40.

The image consists of consecutive frames in time. For example, if the frame at the current time t is the current frame, the frame at the immediately preceding time t-1 is referred to as the previous frame, and the frame at the time t + 1 is referred to as the next frame. On the other hand, each frame has a color image (or a color image) and a depth image (or depth information).

That is, the depth image 61 and the color image 62 are composed of consecutive frames in time. One frame has one image. Also, the images 61 and 62 may have one frame (or image). That is, the images 61 and 62 correspond to one image.

Detecting a face in a depth image and a color image means detection in each of a depth / color frame (or image), but the term "image" is used below unless there is a need for a special distinction.

First, the overall face detection and tracking method used in the present invention will be described with reference to FIG.

In general, face tracking must determine a target face to be tracked, and a face detection method [Non-Patent Document 17] is mainly used for this purpose. Face detection is usually time consuming because the face must be searched for the whole image if the input image is given. Therefore, it is not easy to track face by applying face detection method to every image frame. Therefore, it is common to detect the face at the beginning of face tracking and track the detected face to the target face.

FIG. 2 shows the relationship between face detection and face tracking considered in the present invention. In the conventional face detection method, the texture image is mainly used. However, since the tracking method according to the present invention uses only depth information, depth information is added as an input of face detection in the drawing.

When a face is detected in the detection step without using depth information, the depth map, position, and size of the face are extracted and used as a template in face tracking. Once a face is detected, only the face tracking is performed from the next frame, and the template of the face tracked every frame is updated.

If the tracking fails, the tracking process can not proceed anymore. In this case, go to the face detection step again and detect the face again. Generally, if the tracking fails, the person who is tracking moves out of the image or changes the scene.

In the present invention, since the face tracking method is proposed, the face detection method is not mentioned, and a conventional method (non-patent document 16) is used.

On the other hand, the use of depth images to track faces is to track irrelevant changes in illumination. The conventional method tracks using RGB or RGB. In this case, if the illumination of the current frame and the next frame change, an error may occur in template matching. However, depth images are used and can be tracked irrespective of changes in lighting, and only the adjustment of the depth value is required for forward and backward movement.

Next, a fast matching method based on template matching according to an embodiment of the present invention will be described with reference to FIG. 3 to FIG.

As shown in FIG. 3, the template matching based fast face tracking method according to the present invention includes: (a) updating a template size (S10); (b) setting a search area in the current frame (S20); (c) performing template matching (S30); And (d) repeating template matching with a template whose size and position have been updated (S40).

4 is a pseudo code representation of the face tracking method of the present invention. As shown in FIG. 4, the face tracking method of the present invention inputs a depth image sequence and a template, and outputs an updated template for each frame of an input image sequence. As a template, use the depth image, position, and size corresponding to the face to be tracked. Each step is described in detail below.

First, the template size and search range are reset (S10, S20).

That is, the size change of the face tracked in the next frame (or the current frame) compared to the previous frame is predicted, and accordingly, the size of the template is adjusted and the search area to be template-matched is adjusted as a result.

The size resetting step S10 of the template will be described in detail.

If a person moves up or down or side to side, the size of the face is almost unchanged, but if moved back and forth, the size will change. Therefore, the size of the template should be updated in accordance with the size of the face of the frame every frame.

Since the relative size of an object in an image is affected by its depth, we first describe the relationship between depth and object size. In general, a depth camera provides the actual depth value, that is, the distance between the camera and the object. The maximum depth of the distance that the camera can be measured, "when the actual distance z and Z represent defines the minimum distance to the _{R z, min, R z, max,} respectively, and n- bits, the distance (Z), the formula (1) and Have the same relationship.

[Equation 1]

However, the closer the depth image we usually touch, the larger the value. The depth map (Z) of this method should be expressed by inverting Equation 1 as shown in Equation (2).

&Quot; (2) "

Therefore, when the actual size of the object is s, the actual distance is z, and the focal length of the camera is f, the relationship between the object size S _sensor and z or Z in the camera image sensor is given by Equation 3.

&Quot; (3) "

If the pixel size of the image sensor is P _sensor , the number N of pixels in the actual image can be expressed by Equation 4.

&Quot; (4) "

If the object size is N when the object is at the depth Z, the graph shown in FIG. 5 can be obtained. Here, the dotted line is the result of Equation 4, the dotted line is the measurement result, and the solid line is the trend line of the points. Here, the error between the dotted line and the measured value or the trend line is due to the error in the measurement and the error in the decimal unit in Equations 1 to 4.

In order to reset the template size of the current frame according to FIG. 5, the depth of the face in the previous frame, that is, the depth of the template generated in the previous frame is compared with the depth of the corresponding current frame to update the size of the template.

First, the template (the template of the previous frame) is divided into m × n subblocks SB in FIG. Then, the average depth of each block (a × b resolution) is determined, and the largest depth is defined as the depth Z _T of the template. SB ^T _{i , j} means the average depth of (i, j) subblocks of a × b resolution.

&Quot; (5) "

In the current frame to be tracked (or current frame), the partial depth image of the position and size corresponding to the template is extracted and the maximum average depth value is obtained through the same process as in Equation 5. This is called Z _T '.

In this case, the size S ^X _{ZT '} of the template to be used in the current frame can be obtained from the size S ^X _ZT of the template obtained in the previous frame as shown in Equation 6. Where X represents the width (hor) or height (ver).

&Quot; (6) "

Here, the number of blocks (m × n) divided by the template is determined in advance through experiments and the like.

Next, the search area adjustment step (S20) will be described.

The area to be searched for template matching is related to the movement speed of the face. In the present invention, it is assumed that a person to be tracked is watching forward, such as watching a TV or a movie. Therefore, in this case, the size of the search area is determined in consideration of the maximum movement speed that can be moved while viewing the image.

The speed of movement, that is, the amount of movement, varies with the depth of the object. That is, the degree of motion on the screen is relatively determined according to the distance (depth) of the object to the camera. Therefore, in the present invention, as shown in FIG. 6, a region expanded relative to the size of the template adjusted according to the distance, that is, the template region is expanded to a certain ratio.

Here, the template uses the size (S ^X _{ZT '} ) of the adjusted template as described above. In the present invention, the template is expanded in the same ratio (?) (Or expansion ratio). The alpha value is a coefficient obtained in the experiment or the like, and is predetermined in advance.

Next, the template matching step S30 will be described.

This is a template matching process for finding the correct face position by searching for each position in the template and its corresponding search area. When template matching is performed with an RGB image, as shown in [Non-Patent Document 16], it may happen that a face can not be found due to illumination change and another portion is searched. Thus, in the present invention, template matching is performed using depth information that is resistant to illumination change.

We use a sum-of-absolute differences (SAD) per pixel in Equation 7 as a cost function for template matching. That is, all positions (i, j) in the search range are searched using Equation 7, and the position having the smallest PSAD _{i, j} value among them is selected as the position (SP _{Opt in} Equation 8) where the template is matched.

&Quot; (7) "

&Quot; (8) "

In Equation 7, k is the number of pixels in the template, T _w and T _h are the width and height of the template, respectively. And D _SA (i + h, j + v) and D _T (h, v) represent the pixel values of the template and the search area.

Finally, the matching position is updated to the new position of the template, and the next frame is set as the current frame, and steps S10 to S30 are repeated (S40). As described above, if the matching position is not found, the face is detected again and a new template is extracted, and the above process is repeated.

Next, a detailed method for improving the speed of the template matching step S30 will be described in more detail.

Preferably, at the time of template matching, PSAD is calculated for each position in the search area, and if the calculated PSAD satisfies a specific condition, template matching is terminated early.

Equation (7) searches all positions of the search area with the PSAD value, and the time depends on the size of the search area. Considering the accuracy of face tracking, it is advantageous to set the search area large. Indeed, to some extent, the larger the search area, the smaller the tracking error (described in more detail below).

However, a sufficiently large search area may not be able to be tracked in real time due to excessive search time. One of the objects of the present invention is to track faces at high speed. Therefore, the early termination method is used to reduce the search time. The background of this method is that the purpose of tracking the position is often to allow a degree of approximation rather than tracking it without any errors.

In order to use the early termination method, it is necessary to judge that the position is tracked approximate. Since the PSAD is used as the cost function, the search is terminated early when the threshold value (T _ET ) of the predetermined PSAD is smaller than T _ET . This can be expressed as Equation 9.

&Quot; (9) "

Further, preferably, the search proceeds in a spiral manner in the search area at the time of template matching.

In the case of the environment set in the previous case, there are many cases in which the person does not move much more than when the person moves, and there are many small movements rather than large movements. Therefore, if you consider early termination, it is more efficient to search around the location of the previous template rather than the usual way (such as raster scan or zigzag scan).

Therefore, in the present invention, a helical search method is used in which the search starts from the position of the previous template and is gradually distant toward the spiral. In FIG. 7, a spiral search is performed when the search area is 5 x 5 [pixel ² ]. Each block in Fig. 7 represents a pixel, the number of each block represents the order of the helical search, and the '0' position is the position of the previous template.

Further, preferably, a sparse search method is applied in template matching.

A sparse search method is applied in the present invention in order to further reduce the search time in the early termination method. In this method, the pixel to be searched next to the currently searched pixel in the search area is not searched for the adjacent pixel (in the case of the spiral search, the next pixel in the search sequence) do. That is, the pixels in between are not searched.

In FIG. 7, an example is shown in which the interval is '3'. The pixels searched in this example are {0, 3, 6, 9, 12, 15, 18, 21, 24}. The sequence searched in this sparse search is shown in Equation 10. Where I represents the interval. If I = 1 in this equation, it means that all regions are searched. The I value is also determined in advance through experiments and the like.

&Quot; (10) "

In another embodiment, a sparse sparse search is performed while a cell to be searched for is regularly determined. Also, as the distance from the center increases, the search pixel may be sparsely caught. Since it is highly probable that the motion is small, it is desirable to make the rare search interval smaller when searching for a position close to the previous position, and to make the search interval longer as the distance becomes larger. In this case, the search speed can be significantly improved.

Next, preferably, when a rare search or early termination method is applied, matching correction is performed to reduce tracking errors.

A correction process is performed to reduce tracking errors caused by sparse searching and early termination, which is to perform an additional search on surrounding pixels of a prematurely terminated pixel.

Two are considered according to the number of additional search pixels, the first of which is a 2-pixel refinement. In this correction method, the telephone number and the posterior pixel of the pixel terminated early in the search sequence are further searched.

The second method is 4-pixel correction (4-pixel refinement), which is a method of further searching for four-directional adjacent pixels of an early-terminated pixel. FIG. 8 shows an example of the spiral search of FIG. 7, where the case of early termination at the pixel 6 is shown. Here, the 2-pixel correction is further searched for {5, 21} pixels and the {5, 7, 19, 21} pixels are further searched for the 4-pixel correction.

The final SP _Opt is determined as the pixel having the smallest value among the PSAD values of the pixels terminated prematurely and the pixels searched for correction in both of the corrections.

Next, a method for determining parameters for face tracking will be described.

The parameters used in the face tracking method described above are determined experimentally. The table of Figure 9 shows the characteristics of the test sequences used for parameter determination. The sequences are self-created using Kinect released by Microsoft, and thus have depth information and color information of 640 × 480 resolution.

In the table of FIG. 9, 'LR' is used to measure left and right movement, 'UD' is used to measure up and down movement, and 'BF' is used to measure back and forth movement. Movements in each direction are made at a speed sufficiently high to accommodate the environment previously set. FIG. 10 shows images of representative three frames of each sequence.

First, an experiment for determining the size of the template division for template re-adjustment will be described.

In order to estimate the size of the template as the face size of the current frame, the template is divided into m × n blocks. Experiments are carried out on various partitioning methods. Since there are too many ways to divide by the combination of m and n, we test from 3 × 3 block to 11 × 11 block with 'm = n'.

In the experiment, the error (TRE (%)) is measured by comparing the size of the reference image (RI ₃ ) and the resized template (RT ₃ ). Equation 11 shows a method of calculating an error. The reference image uses a face image detected using the Adaboost algorithm (Non-Patent Document 17).

&Quot; (11) "

The experimental results are shown in Fig. 11, and it can be seen that the error gradually increases from the 3x3 block to the 11x11 block. Therefore, in the present invention, the template division is defined as a 3x3 block having the smallest error.

Next, an experiment for determining the range of the search area, particularly, the expansion ratio? Will be described.

In the previous section, we set the search area to a relative value of 100 + 2α (%) in relation to the template size. Here, the expansion ratio α is determined through experiments.

For each test sequence, tracking error is measured while varying α from 15% to 50%. At this time, the tracking error is obtained using Equation 12.

&Quot; (12) "

Here, T _SP (Template _{SearchPosition} ) refers to the position of the face tracked at α (%), and T _BP (Template _{Best Position} ) refers to the position of the face found with the entire image as the search range. And T _DS (Template _Diagonal Size) refers to the diagonal size of the template.

In FIG. 12, experimental results according to an embodiment are shown. It can be seen that the tracking error of the 'UD' sequence is higher than 20% at α <21. The reason for this is that the search range is so small that the face is out of the search range, and relatively 'BF' has a low error throughout the reason, because 'BF' . Considering the three sequence modes, the search area is determined to be α = 41 (%) where the tracking error of all sequences is almost 0%. Preferably, the expansion ratio alpha is determined so that the tracking error is less than 1%.

Next, an experiment for determining the PSAD threshold for early termination will be described.

Previously, an early termination technique was described to reduce execution time. Here, the threshold used in the early termination method is determined experimentally. FIG. 13 shows the result of measuring the tracking error and the execution time (Time) while changing the threshold value (T _ET ) from '0' to '5'.

As shown in FIG. 13, as the threshold value increases, the tracking error increases, but the execution time decreases. In the present invention, since the target of high-speed tracking is set, the threshold T _ET = 2, in which the execution time is less than 40 ms, is set to 2, and the tracking error by the figure is less than 4%.

That is, preferably, the threshold value that is less than the target execution time is determined as the PSAD threshold based on the target execution time.

Next, an experiment for determining the interval for the sparse search will be described.

In order to reduce the execution time as well as the early termination, we proposed a sparse search method. At this time, preferably, the interval is determined through experiments.

The experiment measures the tracking error and the execution time while increasing the interval I from '1' to '5'. FIG. 14 shows an experimental result according to an embodiment. In this experiment, T _ET = 2 is performed as previously defined. As shown in FIG. 14, as the interval increases, the tracking error increases and the execution time decreases. Therefore, I = 3 with less than 5% tracking error and less than 30 ms average tracking time at optimal intervals.

Next, the effect of the present invention through experiments will be described in more detail with reference to Figs. 12 to 20. Fig.

Experiments were conducted to apply the parameters obtained through the face tracking method and experiment according to the present invention to various test sequences. The implementation uses Microsoft Visual Studio 2010, OpenCV Library 2.4.3 in the Microsoft window7 operating system, and the specifications of the PC used in the experiment were Intel Core i7 CPU of 3.40GHz and 16GB of RAM.

The test sequences used to determine the parameters above are somewhat different from realistic situations because they were produced by extreme movements for parameter extraction. Therefore, a more realistic test sequence was created and used in-house.

15 shows the characteristics of the sequences used in the table, 'WL' and 'S & J' are self-produced sequences, and Lovebird1 is a multi-point test sequence provided by MPEG. 'WL' is a sequence in which a person sits and moves freely in front and rear, left and right, and up and down, and 'S & J' is a sequence in which two people move freely and one person leaves the screen and one person remains.

The face tracking method according to the present invention and the results of applying the extracted parameter values to these sequences are shown in the table of FIG. This table also includes FS (Full search) with no early termination and sparse search, Early Termination (ET) with early termination only, Early termination and sparse searching, and 2-pixel refinement, and 4-PR (4-pixel refinement), respectively. The average tracking error (%) and the average execution time (ms) are shown. As shown in the table, the 4-PR has the lowest tracking error (%), but the execution time is the fastest when applying the early termination only. However, since 2-PR and 4-PR do not make much difference in speed and are sufficient for real-time operation, 4-PR may be selected as the final algorithm.

Figure 17 shows the resultant image tracked in each sequence. As shown in the figure, it can be seen that the method according to the present invention tracks only the person closest to the camera when there are two or more people (Fig. 17 (e) and (f)). You can also see the side looking at the face without looking at the front, because it targets depth images, not color images.

The table of FIG. 18 was compared with the conventional method. [Non-Patent Documents 15] and [Non-Patent Document 19] do not have experimental result information for the same comparison with the present invention. Therefore, in the present text, the information of the present invention is modified for the sake of an equal comparison. In the table of FIG. 16, 2-PRs having middle error and speed among ET, 2-PR, and 4-PR are selected and compared. Depth + RGB is input as the sequence, but only the depth information is used in the algorithm according to the present invention. As shown in Table 4, the method according to the present invention shows excellent results in accuracy and speed as compared with [Non-Patent Document 15]. In [Non-Patent Document 19], the speed was not shown, and therefore, the method according to the present invention showed a worse result in the case of a resolution of 640x480 and a better result in a case of 1024x768 It is showing. At 640x480, the less accurate results were achieved with a fast tracking algorithm of 404 frames per second, so if you adjust the parameters (T _ET = 2, I = 3) to increase the accuracy, .

In the present invention, a method of tracking a face at high speed using only depth information has been described. The template matching method is used as a basic method for face tracking and the early termination method and the sparse search method are used to reduce the excessive amount of computation which is a problem of the template matching method. Matching correction was performed. In order to correct the depth change due to the back and forth movement, the depth value of the face to be tracked was estimated and the size of the template was appropriately adjusted according to the result. Also, we adjusted the search area to perform template matching according to the size of the adjusted template.

In the present invention, values are obtained using test sequences in order to derive the parameter values of the proposed face tracking method. Then, we experimented using the self-generated sequence and MPEG sequence for the experiment, not the test sequence, using the obtained parameters. In the experimental result, the tracking rate was 100%. In the self-produced sequences 'WL' and 'S & J' using Kinect, the tracking error was about 3% and the execution time was 2.45 ms. In the multi-point sequence Lovebird1 provided by MPEG 1% tracking error and 7.46ms performance time.

The method according to the present invention confirms that there is a complementary relationship between the threshold value for early termination, the tracking error according to the interval for sparse search and the execution time as expected. Therefore, the face tracking method according to the present invention is suitable for use in a real time face tracking system of 30 frames per second or more, and can be used in various fields by utilizing the complementary relationship between the execution time and the tracking error.

Although the present invention has been described in detail with reference to the above embodiments, it is needless to say that the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the spirit of the present invention.

10: Face 20: Kinect
21: depth camera 22: color camera
30: computer terminal 40: program system
61: depth image 62: color image

Claims (9)

A template matching-based high-speed image processing unit that receives a depth image and a color image having frames consecutive in time from a camera that photographs a face of a person, detects the face in one frame to generate a template, In the face tracking method,
(a) comparing a depth of the template with a depth of a current frame to update a size of the template;
(b) setting a region of the current frame extending to a predetermined ratio (hereinafter referred to as expansion ratio) of the size of the updated template as a search region, with the updated template as a center;
(c) performing template matching in the search area with the updated template and selecting a matching position; And
(d) repeating the steps (a) to (c) by updating the matched position to the position of the updated template and using the next frame of the current frame as a current frame,
In the step (c), the search is progressively progressed in a spiral direction starting from the position of the search area corresponding to the position of the template,
In the step (c), template matching with the updated template is performed while sequentially moving from the search area to a neighboring location, and when shifting from the position of the searched pixel to the position of the next search pixel, The next pixel is selected and searched beyond the pixel at a predetermined interval,
In the step (c), if the pixel at the matched position is selected, further matching is performed with respect to the pixel at the selected position and the neighboring pixel, and correction is performed for the upper, lower, left, right, Speed face tracking based on template matching.
The method according to claim 1,
In the step (a), each of the template and the region corresponding to the template in the current frame (hereinafter referred to as a sampling region of the current frame) is divided into subblocks, Wherein the size of the template is adjusted and updated in proportion to a depth difference between the template and the sampling area.
The method according to claim 1,
In the step (c), the updated template and a sum-of-absolute differences (SAD) per pixel at each position in the search region are obtained, and a position having a minimum value is selected as a matched position A high speed face tracking method based on template matching.
The method according to claim 1,
Wherein the search is prematurely terminated when the updated template and the SAD (PSAD) per pixel in the search area are smaller than a predetermined threshold in step (c).
delete delete delete delete A computer-readable recording medium having recorded thereon a program for performing the template matching based high-speed face tracking method according to any one of claims 1 to 4.

Images