JP2003346158A - Face area tracking method by face image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face area tracking method by a face image capable of withstanding the deformation of a pattern, providing an excellent follow-up performance, and withstanding image noise by updating templates at proper timing since it is hard to accurately track the face area when the deformation of the face area pattern is increased. <P>SOLUTION: In this face area tracking method by the face image, the direction of the face of a person is tracked by comparing the plurality of templates with different sizes of corresponding areas with the inputted image, the relative position of a position on an input image where the similarity thereof to the large template becomes maximum to a position on the input image where the similarity thereof to the small template becomes maximum is obtained (9). When the amount of displacement from the relative position of the large and small templates at previous updating of the templates becomes larger than a specified amount (7), the templates are updated to accumulate the templates equivalent to the input image (6, 8). <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a face area tracking method using a face image, which performs a robust, high-accuracy, high-speed face area tracking of a person's face area in an image.

[0002]

2. Description of the Related Art An active interface using a face movement is useful when a hand is inconvenient or when a hand is occupied by luggage or the like. As a means of human communication in normal life, in addition to words and gestures, there is a way to express affirmation by waving the face vertically, or send a signal with the face turned in that direction to indicate something . This indicates that such an input support method based on a gesture of moving a face is also useful for an input interface of a computer.

[0003] If the movement of the face can be captured, it can be used not only as an active operation method but also for the interest of the person.
It can also be used to understand the target of attention, preferences (market research), and mental status.

Here, in order to recognize the movement of a face, a face must first be detected from an image and the detected face must be tracked. In the past, there were problems such as restrictions on the position of the face, the number of cameras, slow speed, and low detection accuracy, and it was difficult to accurately track fast face movements.

On the other hand, there is a method of detecting a head movement without tracking a face. This is a method to detect the three-dimensional movement of the head based on the contrast between the face and the hair (“Head Reader: Real-time Detection of Head Movement Using Images” IEICE Transactions, Vol.J74-D-II , No.3, pp.398-406, March 1991).
However, this method is susceptible to changes in lighting conditions because it uses the shading of the image, and requires a background image that does not change. In addition, it is affected by whether the hair is hanging near the ears.

There is also a method of using a stereo camera to manually register feature points of a face and tracking the feature points by normalized correlation matching to obtain a face direction and a gaze direction (“An a
lgorithm for real-time stereo vision implementatio
n of head pose and gaze direction measurement ”. I
n Proc. of the Int'l Conf. on Automatic Face andGe
sture Recognition, 2000). However, a plurality of cameras are required, and the influence of the deformation of the feature point pattern due to the change in the face direction is not considered.

[0007] There is a method of preparing a face template for each face orientation angle and detecting faces corresponding to various face orientations ("View-based and modular eigenspaces for face res").
cognition ”. In Proc. IEEE Conf. on Computer Visio
n and Pattern Recognition, p.84, 1994). However, a face image taken at a preset angle must be used as the face direction angle, and a template cannot be created by classifying each face direction angle from the face image itself.

[0008] A method of tracking an object to be tracked using a template is mainly based on a raster scan type pattern search that sequentially scans the entire image sequentially from the upper left to the lower right (for example, by Akio Okazaki, "First Image Processing Technology"). ”, Industrial Research Council, 2000). Furthermore, a method of tracking or searching for a pattern while changing an enlargement magnification (reduction magnification) when associating a template with a target image has not been established. This is one of the reasons why it takes a very long calculation time to track the face area.

[0009]

SUMMARY OF THE INVENTION The present invention has been made in consideration of the above circumstances, and a first object of the present invention is to update a relative position of a large and small template when the relative position change becomes large. Update the template at an appropriate timing when the pattern deformation of the face area becomes large, robust to pattern deformation and excellent in following performance,
An object of the present invention is to provide a face area tracking method using a face image that is resistant to image noise.

A second object is to create a template classified by face direction angle from the face image itself while using various face images whose face direction angles are unknown, and to switch the template appropriately. It is an object of the present invention to provide a face area tracking method using a face image that is robust to pattern deformation and has excellent follow-up performance, and that is resistant to image noise, which enables face area tracking to be performed.

A third object is to change the value of (x, y, r, φ) in the previous tracking result in a space spanned by the position, magnification, and rotation angle (x, y, r, φ) of the template. It is an object of the present invention to provide a high-speed and accurate face area tracking method by performing tracking by searching such that the density of search points is a dense distribution at the center.

[0012]

According to a first aspect of the present invention, there is provided an image processing apparatus, comprising: comparing a plurality of templates having different sizes of corresponding regions with an input image; A face area tracking method based on a face image for tracking a person's face area, wherein a position on the input image at which the similarity with the large template is maximum and a position on the input image at which the similarity with the small template is maximum When the amount of displacement from the relative position of the large and small templates at the time of the previous template update becomes larger than a predetermined value, the template is updated and the template corresponding to the input image is stored. A face area tracking method using a face image is provided.

The present invention provides a face area tracking method using a face image that is robust to pattern deformation and has excellent follow-up performance and is resistant to image noise by controlling the update of the template using the positional relationship between the large and small templates. Can be provided.

Next, a second aspect of the present invention is to provide a face image based on a face image which tracks the orientation of a person by comparing the input image with a plurality of templates having different sizes of corresponding areas. An area tracking method, wherein a position on the input image at which the similarity with the large template is maximized;
Finding the relative position to the position on the input image that maximizes the similarity with the small template, and tracking the face area using multiple face orientation templates prepared according to the size of the relative position The present invention provides a face area tracking method using a face image characterized by:

The present invention uses such a map of a face template (face template map) depending on (x, y) to select a template in consideration of the deformation of the face pattern due to a change in the face direction. A face area tracking method using a face image that is robust to deformation and excellent in follow-up performance and resistant to image noise can be provided.

Next, according to a third aspect of the present invention, a template having a predetermined number of pixels is compared with an input image.
A face area tracking method based on a face image for detecting a degree of coincidence between an image and a template, wherein a position on an input image corresponding to a template is set to (x, y), and an area of an arbitrary size on the image is determined. The magnification of enlargement or reduction of the template for associating the template is r, the rotation angle of the template for associating the template with the target area rotated on the image is φ, and (x, y , R, φ), the (x, y) template is displaced so that the search points are densely distributed at the center, or the template with r, φ varied is displaced. Provided is a face area tracking method using a face image, which is characterized by tracking an area.

The present invention can provide a high-speed and accurate face area tracking method by performing tracking in accordance with such a dense distribution centered on the previous position.

[0018]

Embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) In this embodiment, a face area tracking method using a face image will be described. FIG. 1 is a configuration diagram showing an example of a face area tracking method using a face image according to a first embodiment of the present invention and a system for realizing the method. FIG. 2 is a conceptual diagram illustrating a face template map according to the embodiment. FIG. 3 is a flowchart illustrating the flow of the face area tracking process according to the embodiment.

The major feature of the face area tracking system 1 shown in FIG. 1 is that the system has a face template map storage unit 6 therein and has a function of learning and updating the face template map sequentially. The face area tracking system 1 includes an image input unit 2, a front face detection unit 3, and a front face dictionary storage unit 4.
And a front face evaluation unit 5 and a face template map storage unit 6
, A template setting unit 7, a tracking template information storage unit 8, and a template tracking unit 9.
By providing such components, when an image is input, the face area tracking system 1 outputs information such as the position of the face area, face orientation information, the result of the face area tracking calculation, Build a face template map inside the system by updating.

The image input unit 2 receives an image of the subject taken by a video camera or the like from outside the system, converts the image into digital electronic data for easy image processing,
The electronic data is sent to the front face detection unit 3 and the template tracking unit 9.

The front face detection unit 3 receives a template for front face detection from a front face dictionary storage unit 4 storing a template for front image detection of a person, and stores an image input from the image input unit 2 By comparing with the template, a face image of a person is detected from the input image. For example, it is possible to use a method in which an image of the eyes and nostrils of a human image is detected by pattern matching and a circular separation filter, and face detection is performed by using the pattern matching and the whole face pattern matching. In this case, the template is a dictionary pattern of the subspace method corresponding to each of the face, eyes, and nose.

The front face evaluation section 5 receives the face image of the person and the position information of the detected face area from the front face detection section 3,
When the front face template 13 is received from the face template map storage 6 and the data of the detected face area is more suitable as the front face template 13, the front face template 13 in the face template map storage 6 is used.
To update. The position information of the detected face area is sent to the template setting unit 7.

The determination as to whether or not it is more suitable as a front face template can be made based on, for example, the similarity value of template matching, the arrangement of the coordinates of eyes and nostrils, and the like.

As shown in FIG. 2, the face template map storage section 6 stores large and small face templates that differ depending on the face orientation angle in association with upward and sideways angles. Make up. Here, the large face template is an image template of the entire face including characteristic parts of the face such as the eyes and the mouth.
The small face template is a template of an image including a characteristic part (for example, a nose) of a face that can be detected three-dimensionally.

In the face template map stored in the face template map storage unit 6, for example, in the initial state, each template for each face direction angle is blank (blank).
First, a large and small face template is created using the image determined to be suitable for the front face image by the front face evaluation unit 5, and the front face template 13 is generated.
Next, the face template other than the front face is learned and updated as the face images of various face directions are input by the operation of the template setting unit 7.

The template setting section 7 includes a front face evaluation section 5
, The tracking template 22 to be used for the next face area tracking is determined and stored in the tracking template information storage unit 8.

Here, the template setting section 7 determines whether or not to register the detected face area as a new face template based on the relative position information of the large and small face templates. This is because, for example, when the face direction changes, such as when the subject turns sideways, the matching position of the small face template shifts in the face direction (for example, the horizontal direction), and the degree of the shift increases as the face angle increases. (Relative position from the front position) is used. When the shift amount becomes larger than a certain threshold value, the detected face area is registered in the face template map storage unit 6 as a face template such as left, right, up and down, as necessary.

A face template map can be constructed using a plurality of templates of three or more sizes, and face area tracking can be performed. When using templates of three different sizes (large, medium, and small) as shown in FIGS. 9, 10, and 11, for example, the position of the face template (middle) with respect to the face template (large) and the position of the face template (middle) The position of the face template (small) is related to the size of the face orientation angle, and based on the comprehensive judgment based on these values, the detected face area is used as a face template for left, right, up, etc. Register in the storage unit 6. As a method of making a comprehensive judgment, an addition sum of magnitudes of shifts of relative positions can be used. By increasing the number of templates in this way, more stable and robust registration processing and face area tracking can be performed.

The tracking template information storage section 8 receives a face template for tracking from the template setting section,
The tracking template 22 used for the tracking process of the face direction which is sequentially updated is stored, and the tracking template 22 is sent to the template setting unit 7 and the template tracking unit as needed.

The template tracking section 9 receives the input image from the image input section 2 and stores the tracking template information storage section 8
And performs a face area tracking process by comparing the input image with the tracking template. The tracking processing result (success / failure of face area tracking, size, position, face direction (angle), and the like of the face area) is output, and information on the face area is sent to the template setting unit 7.

In the tracking process, as shown in FIG. 4, a tracking template 22 consisting of, for example, 9 pixels × 9 pixels = 81 pixels is made to correspond to a certain position on the input image area 21 and the similarity is calculated by the template matching process. Is achieved by changing a plurality of search positions and magnifications. Then, when the maximum similarity exceeds a predetermined threshold, it is determined that the face area is detected at the position and the magnification.

The magnification indicates an enlargement / reduction ratio of an image. If the number of pixels of the image area corresponding to the template is 9 × 9 pixels, the magnification is exactly 1 ×. If the number of pixels is n × n, the magnification is n / 9.
Double. When the magnification is not 1, the image can be made to correspond to each pixel of the tracking template by enlarging or reducing the image. Here, the number of pixels of the tracking template is set to 9 × 9, but any number may be set as appropriate, depending on the problem.

The search for the image position and the magnification in the tracking process is limited by the calculation time, and in practice, it must be performed sparsely at a certain interval, but in this embodiment,
When the position is determined with respect to the values shown in FIGS. 5, 7, and 8, and the magnification is determined with respect to the value illustrated in FIG. 6, the search process can be performed more efficiently.

Specifically, when the previous position is (x, y), the vicinity of the previous position is searched for in detail with reference to the previous position, and the search process is roughly performed as the search position moves away. For example, (x, y) of the input image is searched using the tracking template (0th search), and then (x ± 1,
y), (x, y ± 1), (x + 1, y ± 1), (x−
1, y ± 1) (first search), and then (x ± 3,
y), (x, y ± 3), (x + 3, y ± 3), (x−
3, y ± 3) (second search) is continued until the n-th search.

Regarding the search for the magnification, specifically, when the previous magnification is r, an image near the previous magnification is generated based on the previous search magnification to search in detail. For example, using a certain small constant a (0 <a <1), first, an input image is searched using a tracking template of r times size (0 th
Search), then search using a tracking template with a magnification of r * (1 ± a) (first search), and then r * (1 ± 3a)
(Second search) is continued until the m-th search.

As for the search for the rotation, a detailed search is performed using a template having a rotation near the rotation angle with reference to the previous rotation angle, and a coarser search is performed as the rotation displacement from the rotation angle increases. For example, if the rotation angle of the previous template is α, the input image is searched (0-order search) using the tracking template that has been subjected to the rotation of α, and then the rotation of α (1 ± a) is performed. (Primary search) using the tracking template thus obtained, and α (1 ±
The search (secondary search) is continued using the rotated tracking template of 3a).

As described above, by performing a search in which the center is dense centering on the previous parameter value, rough but high-speed tracking is performed for a fast movement of the face, and a slow movement of the face is performed. For this, precise tracking can be performed.

FIG. 3 is a flowchart showing a tracking procedure of the face area tracking method according to the present embodiment.

First, it is determined whether or not to perform face area tracking based on an input from the subject or the system (ST).
1) If not, the process ends. When performing face area tracking, an input image from the image input unit is processed to perform front face detection and template tracking (ST2). Next, it is determined whether or not the previous face area tracking processing was successful (ST
3) If the previous time was successful, the process proceeds to step ST8; if the previous time was unsuccessful, the process proceeds to step ST4.

In step ST4, the front face detection section 3 performs a front face detection process to determine whether a front face has been found (ST5). If a front face has been found, the process proceeds to step ST6. Return to ST1.

In step ST6, the front face evaluation unit 5 performs a front template updating process, copies the front template to the tracking template 22 (ST7), and the template tracking unit performs a large / small template tracking process (ST7).
8). Next, it is determined whether or not the template tracking has succeeded (ST9). If the template tracking has failed, the process returns to step ST1, but if the tracking succeeds, face position information is output (ST1).
0), according to the magnitude of the relative position change between the large and small templates (ST11).
Are updated, and in any case, the process returns to step ST1.

[0042]

As described in detail above, according to the present invention, by controlling the updating of the template using the positional relationship between the large and small templates, it is robust to pattern deformation and excellent in follow-up performance and resistant to image noise. It is possible to provide a face area tracking method using a face image.

Further, according to the present invention, a template of a template in which the deformation of the face pattern due to a change in the face direction is considered using a map of the face template (face template map) depending on the relative position (x, y) of the large and small templates. By making a selection, it is robust to pattern deformation and has excellent following performance,
A face area tracking method using a face image that is resistant to image noise can be provided.

Further, according to the present invention, a high-speed and accurate face area tracking method can be provided by performing tracking in accordance with a dense distribution centered on the previous position.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing an example according to a first embodiment of the present invention.

FIG. 2 is an exemplary conceptual diagram illustrating a face template map according to the embodiment;

FIG. 3 is a flowchart showing an example of processing of the face area tracking method according to the embodiment.

FIG. 4 is an exemplary conceptual diagram illustrating face area tracking using the tracking template according to the embodiment;

FIG. 5 is a conceptual diagram illustrating a face area position search according to the embodiment according to a dense distribution.

FIG. 6 is a conceptual diagram illustrating a face template magnification search according to the dense distribution according to the embodiment.

FIG. 7 is an exemplary conceptual diagram illustrating a face area position search according to the embodiment according to a dense distribution.

FIG. 8 is a conceptual diagram illustrating a face area position search according to the dense distribution according to the embodiment.

FIG. 9 is a conceptual diagram illustrating a face template map according to the embodiment.

FIG. 10 is a conceptual diagram illustrating a face template map according to the embodiment.

FIG. 11 is an exemplary conceptual diagram explaining the face template map according to the embodiment;

[Explanation of symbols]

1. Face area tracking system 2. Image input unit 3: Front face detection unit 4: Front face dictionary storage unit 5. Front face evaluation unit 6 ... Face template map storage 7 ... Template setting section 8. Tracking template information storage 9: Template tracking unit 11 ... Face template (large) 12 ... Face template (small) 13. Front face template 14 ... Face template (medium) 21: Input image area 22: Tracking template

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Atsuto Maki             No. 1 Komukai Toshiba-cho, Kawasaki-shi, Kanagawa             Toshiba R & D Center F term (reference) 5L096 CA04 FA67 FA69 GA13 HA05                       JA03 JA09 JA16 JA18 KA03

Claims (3)

[Claims] 1. A face area tracking method based on a face image for tracking a face direction of a person by comparing a plurality of templates having different sizes of corresponding regions with an input image, the method comprising: The relative position between the position on the input image at which the similarity of the template becomes maximum and the position on the input image at which the similarity with the small template becomes the maximum is calculated, and the displacement from the relative position of the large and small templates at the time of the previous template update A face area tracking method using a face image, characterized in that when the amount becomes larger than a predetermined value, the template is updated and a template corresponding to the input image is stored. 2. A face area tracking method based on a face image for tracking a face direction of a person by comparing a template of a plurality of sizes having different sizes of corresponding areas with an input image, the method comprising: The relative position between the position on the input image that maximizes the similarity of the input image and the position on the input image that maximizes the similarity with the small template is calculated, and a plurality of face orientations prepared corresponding to the relative position are obtained. A face area tracking method using a face image, characterized in that the face area is tracked using the template. 3. A face area tracking method using a face image for comparing a template having a predetermined number of pixels with an input image and detecting a degree of coincidence between the image and the template. Is set to (x, y), the magnification of the template for making the template correspond to an area of an arbitrary size on the image is set to r, and the template rotated to the target area rotated on the image is set to r. Let the rotation angle of the template to correspond to φ be (φ, y) so that the search points are densely distributed around the center of the value of (x, y, r, φ) in the previous tracking result. A face area tracking method using a face image, wherein a target area is tracked by displacing a template or displacing a template in which r and φ are varied.