JP4043973B2 - Face detection system and method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a face detection system using a face image and a method thereof, which appropriately cuts out a person's face image from an image over a wide range of face orientation angles.
[0002]
[Prior art]
(1) Face detection using face detection images Face detection is a technology that checks whether a human face exists in two-dimensional image data, and if so, finds its position, and can be used in various fields. It is. However, it is difficult to robustly detect faces in various states (various face orientations, various lighting conditions, various facial expressions, etc.) from images taken with a camera in real time using a computer. Even with the state-of-the-art technology, it cannot be said that the performance has been reached yet. For this reason, it seems that some of the application fields of face detection have not yet been realized.
[0003]
At present, the face detection technology is mainly used in a face recognition system as an application to a security system such as a door lock. However, if the performance of face detection technology is further improved and face detection and face direction calculation can be performed for various face orientations, the driver of the vehicle, the audience of the exhibit, the operator of the console terminal, the game Face detection technology can be applied to the operator of the terminal.
[0004]
For vehicle drivers, face detection and face direction calculations can be used to detect side effects that hinder driving and to sound an alarm, or to provide appropriate information for safe driving according to the face direction. .
[0005]
For the audience of the exhibit, it is possible to identify the exhibition that the audience is paying attention by face detection and face orientation calculation, and to automatically present information related to the exhibition.
[0006]
For the operator of the console terminal, it is possible to increase work efficiency by specifying the display window to be operated by face detection and face direction calculation and moving the cursor.
[0007]
For game terminal operators, it is possible to change the viewpoint of the game environment by face detection and face orientation calculation, to set the aim of shooting, and to use it for communication with characters using the swing motion it can.
[0008]
(2) Conventional face detection technology As one of the promising conventional face detection technologies, detection of facial feature points such as eyes and nose using a separability filter and detection of a pattern of a region including the feature points are performed. Some are combined (see, for example, Patent Document 1, Non-Patent Documents 1 and 2). Although this combination method is complicated in processing, the face detection is performed only by feature point detection using a separability filter, or the face is detected only by matching with the pattern of the entire face area. The face detection is more robust and accurate than the method. However, this method sometimes fails to detect a face for a large change in face orientation.
[0009]
As another face detection method, there is a method in which face orientation templates having different face orientations are registered in advance and used (for example, see Non-Patent Document 3). This method can perform face detection even for a large change in face orientation, but the accuracy of feature point detection is not as good as that using a separability filter.
[0010]
Furthermore, in the conventional face detection technique, eyes are used as facial feature points, and face detection sometimes fails when wearing spectacles.
[0011]
[Patent Document 1]
Japanese Patent No. 3279913 [0012]
[Non-Patent Document 1]
Yamaguchi, Fukui, “Robust Face Recognition System“ Smartface ”for Changes in Face Orientation and Expression,” IEICE Transactions, June 2001, Volume J84-D-II, No. 6, p. 1045-1052
[0013]
[Non-Patent Document 2]
Fukui, Yamaguchi, “Facial feature point extraction by a combination of shape extraction and pattern matching”, IEICE Transactions, August, 1997, J80-D-II, No. 8, p. 2170-2177
[0014]
[Non-Patent Document 3]
A. Pentland, 2 others, View-based and modular eigenspaces for face recognition, Proceedings of IEEE Computer Society Society Computer Reputation, United States. 84-91
[0015]
[Problems to be solved by the invention]
The present invention has been made in consideration of the above circumstances, and its purpose is to provide a robust and highly accurate face detection system even when the face direction of a person changes greatly from side to side or when wearing glasses. It is to provide a method.
[0016]
Note that “robust” means “a property that makes it difficult to make a major failure even under adverse conditions” when performing face detection processing or the like using an image. For example, when the face image is deformed, the illumination condition changes, or the face image color changes due to makeup, if the detection process is not likely to fail, it is called a “robust face detection method”.
[0017]
And this invention mainly provides the outstanding technique regarding the robustness of the face detection process when a face greatly turns sideways.
[0018]
[Means for Solving the Problems]
The present invention is a face detection system for detecting a human face from an image, an image acquisition unit for acquiring the image, and a degree of separation detection unit for extracting feature point candidates from the acquired image by a degree of separation filter. A face direction template storage unit storing a plurality of face direction templates, a right nostril template storage unit storing a right nostril template, a left nostril template storage unit storing a left nostril template, and the extracted feature point candidates . a right nostril detecting means for performing right nostril detected by performing template matching by matching the right nostril template the storage, the left nostril performs template matching by matching the left nostril template the stored in the extracted feature point candidate a left nostril detecting means for detecting, frontal and leftward face direction when the right nostril is detected Plates, or the when the left nostril is detected selects the frontal and right direction of the face of the template, performs face detection by performing a template matching the selected face orientation template matching on the obtained image And a face detection unit.
[0026]
The present invention combines a separability filter, right nostril detection using a right nostril template, left nostril detection using a left nostril template, and face detection using a plurality of face orientation templates. Even if the angle changes greatly from side to side, a robust and highly accurate face detection system can be provided.
[0035]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0036]
(First embodiment)
(1) Definition of terms In this specification, terms used as prerequisites are defined.
[0037]
(1-1) The definition of “left and right” is performed.
[0038]
The definition of right and left is based on the anatomical right and left of the subject (mainly a human face). The right eye in a normal (not upside down) face image taken from the front with a normal camera would be on the left side from us toward the face image when we look at the face image. The same applies to the face orientation and nostril. Further, the left-facing face described in FIG. 3 is facing right from the face image from us. On the other hand, the face facing right is facing left from us toward the face image. Further, the right nostrils 23, 33 in FIGS. 2 and 3 are on the left side from the view to us, and the left nostrils 24 and 34 are on the right side from the view to the view. By defining left and right as described above, there is no ambiguity in the definition of the right eye or right direction even if the subject is turned upside down.
[0039]
The right nostril and the left nostril are collectively referred to as “single nostril” for both nostrils.
[0040]
(1-2) Definitions of “face-to-front”, “upward”, “downward”, “rightward”, “leftward”, “upward”, and “lateral” are performed.
[0041]
The appearance of the face image reflected in the camera changes depending on the posture of the head. Therefore, the posture of the head with respect to the camera (that is, the posture of the face) is referred to as “face orientation”.
[0042]
As shown in FIG. 2, the face direction when facing the camera straight to the camera is defined as “front direction”.
[0043]
When facing upward from the front, it is called “upward”, and the greater the angle change, the greater the “upward degree”.
[0044]
When facing down from the front, it is called “downward”.
[0045]
When facing right rather than front, it is called “right”, and the greater the change in angle, the greater the right direction.
[0046]
When facing the left rather than the front, it is called “left”, and the greater the change in angle, the greater the leftward degree.
[0047]
As shown in FIG. 4, the rightward degree and the leftward degree are collectively referred to as “lateral direction degree”.
[0048]
(1-3) Template matching will be described below.
[0049]
“Matching” (see, for example, Non-Patent Document 4) means that when there are two images of the same size, it is determined whether or not they are the same by superimposing them and seeing the difference. “Matching” is also called “collation”.
[0050]
“Template matching” (see, for example, Non-Patent Document 4 “Image Processing Standard Textbook Editorial Committee (Edition), Image Processing Standard Textbook, Association for Promotion of Image Information Education (CG-ARTS), 1997”), “ This means that a standard image called “template” is prepared in advance and it is determined whether or not the target image is the same as the template by matching the target image with this template.
[0051]
As a scale for measuring the difference between two images, for example, the sum of absolute values of differences (SAD: Sum of Absolute Difference) is used. In SAD, a target image having an image size of M × N is I (m, n), (m = 1,..., M, n = 1,..., N), and a template that is a standard pattern prepared in advance is T ( m, n), (m = 1,..., M, n = 1,..., N), the absolute value of the difference I (m, n) −T (m, n) is m = 1,. , N = 1,..., N is obtained by calculating M × N sums.
[0052]
Other measures used include normalized cross-correlation and sum of squares of differences. Here, a scale using a subspace method or an incremental code correlation method is also included. That is, if a matching scale is defined, the case where the template used is not the original image but is converted is included. The scale may be called “similarity” or “difference” in other words. At this time, when the difference between the two images is smaller, the larger value is called “similarity”, and when the difference is smaller, the smaller value is called “difference”.
[0053]
If the image input to the face detection system has a size larger than M × N and it is not known in advance where the target image to be compared is included in the image, move both templates while moving the template in the image. Check the similarity of images. Then, when the maximum similarity is larger than a preset threshold, it is determined that the same template as the template has been detected at a position that gives the maximum similarity. If the maximum similarity is smaller than a preset threshold value, it is determined that the same image as the template has not been detected in the input image.
[0054]
It may be necessary to detect a target image having an image size larger than the image size of the template. At that time, matching is performed by moving the template in the image and simultaneously reducing the image. For example, the face template for detecting the face has an image size of 15 × 15, and the size of the face area in the image is 100 × 100. In this case, both the process of reducing the image at a reduction ratio = 15/100 and the process of moving the template in the image are performed. When the size of the face area is not known in advance, it is necessary to change the reduction ratio, and matching is performed for various values of the reduction ratio. By making the size of the template smaller than the size of the target area in the image, the computer processing time and the memory amount can be saved.
[0055]
In this way, by the template matching process, it is possible to know whether or not the same template exists in the image, and if so, the detected position and size. If it is determined that it exists, it is said to have been detected. The detection position is expressed by a convenient method for subsequent processing, such as the center position of the template when superimposed on the template, the positions of the upper left corner and the lower right corner of the template.
[0056]
(2) Configuration of Face Orientation Calculation System 1 FIG. 1 is a configuration diagram showing the face orientation calculation system 1 according to the first embodiment of the present invention.
[0057]
When an image is input, the face orientation calculation system 1 finds a face from the image and outputs a head position, a nose position, and a face orientation. Inside the system, right nostril detection and a left nostril are detected by a separability filter. It is characterized by having a part for performing detection and a part for performing face detection using a plurality of face orientation templates, and as shown in FIG. 1, a nose detection unit 2, an image input unit 3, a face detection unit 10, The face direction template storage unit 11 and the face direction calculation unit 12 are configured.
[0058]
(2-1) Image input unit 3
The image input unit 3 receives an image input from the camera, stores it, and sends an image to the nose detection unit 2 and the face detection unit 10 as necessary.
[0059]
(2-2) Nose detection unit 2
The nose detection unit 2 receives the image input to the image input unit 3, determines whether or not the nose exists in the image, and sends the determination result and information on the detected nose and nostril position to the face detection unit 10. As shown in FIG. 1, the separation degree detection unit 4, the right nostril detection unit 5, the right nostril template storage unit 6, the left nostril detection unit 7, the left nostril template storage unit 8, and the nose It consists of a detection determination unit 9.
[0060]
The separability detection unit 4 receives an image from the image input unit 3, extracts facial feature point candidates using a separability filter, and extracts feature point candidate information and images from the right nostril detection unit 5 and the left nostril detection unit 7. Send to.
[0061]
The right nostril detection unit 5 receives the feature point candidate information and the image from the separation degree detection unit 4, receives the right nostril template from the right nostril template storage unit 6, and the right nostril exists in the image based on these information. A search is performed to detect the right nostril if present. Then, the right nostril detection result information is sent to the nose detection determination unit 9.
[0062]
Here, the “right nostril template” is a data obtained by converting the image of the region near the right nostril position as it is or by performing principal component analysis, etc., and is used in the image matching by the template matching described above. It is.
[0063]
The right nostril template storage unit 6 stores the right nostril template in advance, and sends the right nostril template to the right nostril detection unit 5 as necessary.
[0064]
The left nostril detection unit 7 receives feature point candidate information and an image from the separation degree detection unit 4, receives a left nostril template from the left nostril template storage unit 8, and a left nostril exists in the image based on these information. A search is made to determine if there is a left nostril. Then, the left nostril detection result information is sent to the nose detection determination unit 9.
[0065]
The left nostril template storage unit 8 stores a left nostril template in advance, and sends the left nostril template to the left nostril detection unit 7 as necessary.
[0066]
Here, the “left nostril template” is an image obtained by converting an image of a region in the vicinity of the left nostril position as it is or by performing principal component analysis or the like, and is data used for image matching by template matching.
[0067]
The nose detection determination unit 9 receives the right nostril detection result information from the right nostril detection unit 5, receives the left nostril detection result information from the left nostril detection unit 7, comprehensively determines these information, and the nose is detected from the image. It is determined whether it has been detected, and the determination result and the position information of the detected nose and nostril are sent to the face detection unit 10.
[0068]
(2-3) Face detection unit 10
The face detection unit 10 receives an image from the image input unit 3, receives a nose detection determination result and detected nose and nostril position information from the nose detection unit 2, and receives a plurality of face orientation templates from the face orientation template storage unit 11. receive.
[0069]
Then, using these pieces of information, the face is detected from the image, and it is determined whether or not the face has been detected comprehensively. The determination result, similarity information for a plurality of face orientation templates, and detection are detected. The position information of the face, nose and nostril thus sent is sent to the face orientation calculation unit 12.
[0070]
Whether or not a face has been detected is determined as follows, for example.
[0071]
If the nose detection determination result is “not detected”, the face is also determined not to be detected.
[0072]
If the nose detection determination result is “detection”, matching is performed by moving the face direction template to a range in the vicinity of the detected nose and nostril position. If the maximum similarity with the face orientation template is greater than the set threshold, it is determined that a face has been detected at a position that gives the maximum similarity. At this time, if only the right nostril is detected, it is considered that the left-facing face is more easily detected than the right-facing face. It can be comprehensively determined that the contradiction is caused by erroneous detection, and it can be determined that the face has not been detected.
[0073]
(2-4) Face orientation template storage unit 11
The face orientation template storage unit 11 stores a plurality of face orientation templates in advance, and sends a plurality of face orientation templates to the face detection unit 10 as necessary.
[0074]
(2-5) Face orientation calculation unit 12
The face orientation calculation unit 12 receives a face detection determination result, similarity information for a plurality of face orientation templates, and detected face, nose, and nostril position information from the face detection unit 10.
[0075]
Then, the face orientation of the detected face is determined based on the similarity information, and the face detection determination result, the position information of the face, nose and nostril, and the face orientation information are output to the outside of the system.
[0076]
(3) Operation Contents of Face Orientation Calculation System 1 FIG. 2 is a conceptual diagram for explaining the positional relationship among the face orientation template corresponding region 21, both nostril template corresponding regions 22, the right nostril 23, and the left nostril 24.
[0077]
The face orientation of this face image is front-facing, and both nostrils are present in the image, and both nostrils can be detected.
[0078]
FIG. 3 is a conceptual diagram for explaining the positional relationship among the face orientation template corresponding region 31, the right nostril template corresponding region 32, the right nostril 33, and the left nostril 34.
[0079]
The face orientation of the face image is leftward, the right nostril is present in the image, and the right nostril can be detected. The left nostril is difficult to see due to hiding, and it is difficult to detect the left nostril.
[0080]
FIG. 4 is a conceptual diagram illustrating the relationship between the degree of lateral orientation of the face and the nostril detectable range.
[0081]
If the face is front, both nostrils and single nostril can be detected, but the right nostril becomes hidden as the degree of lateral orientation turns right, and as a result, only the left nostril can be detected. Furthermore, if the rightward degree increases, the left nostril can no longer be detected. Conversely, the left nostril becomes hidden as the degree of lateral orientation becomes leftward, and as a result, only the right nostril can be detected. Furthermore, when the degree of leftward direction increases, the right nostril can no longer be detected.
[0082]
Thus, in order for face detection to succeed even if the face is facing right or left, both left nostril detection and right nostril detection are performed when detecting the nostril that is a feature point of the face. Thus, it can be seen from FIG. 4 that if the nose is detected by detecting at least one of them, nose detection and face detection using the nose can be performed for a wide range of lateral orientation.
[0083]
(4) Face Orientation Calculation Processing Procedure FIG. 5 is a flowchart showing the processing procedure of the face orientation calculation system 1 in this embodiment.
[0084]
First, an image is input (ST1), the separability detection unit 4 applies a separability filter to the input image, and a plurality of positions on the image where the separability value is large and the maximum value (separation peak) is set. Individually extracted (ST2).
[0085]
Next, the right nostril detection unit 5 collates the right nostril template with the image on the separation degree peak (ST3), and the left nostril detection unit 7 collates the left nostril template with the image on the separation degree peak (ST4). These steps ST3 and ST4 can be processed simultaneously.
[0086]
Next, the nose detection determination unit 9 determines whether or not the nose is detected from the image (ST5). If it is detected, the process proceeds to step ST7, and if it is not detected, the process proceeds to step ST1 (ST6).
[0087]
In step ST7, a face search range for moving the face orientation template is set based on the detected nose position information. The face search range may be set to a range that does not significantly impair the consistency between the detected nose position and the face position, or may be limited to only one face area determined based on the nose position. is there.
[0088]
Next, the face detection unit 10 detects a face from the image, and the face direction calculation unit 12 determines the face direction of the face (ST8).
[0089]
Next, the face orientation calculation unit 12 outputs the head position, the nose position, and the face orientation (ST9). When the face orientation calculation is continued, the process goes to step ST1, and when the face orientation calculation is finished, the face orientation calculation is finished. (ST10).
[0090]
(Second Embodiment)
FIG. 6 is a block diagram showing a face orientation calculation system 101 according to the second embodiment of the present invention. The same parts as those in FIG. Only the differences are described.
[0091]
In this embodiment, the detection performance for a face facing forward is improved by utilizing both nostril detection.
[0092]
In particular, when the threshold value for detecting the right nostril and the left nostril is set higher than the threshold value for detecting both nostrils, a single nostril of the right nostril or left nostril cannot be detected, but both nostrils may be detected. Therefore, in such a case, it is effective to detect a face facing forward using the detection state of both nostrils. The reason for setting the threshold for detecting the right nostril and the left nostril higher than the threshold for detecting both nostrils is because both nostrils are unique feature point candidates in which two holes are arranged. This is because it can be detected.
[0093]
(1) Configuration of Face Orientation Calculation System 101 The nose detection unit 102 includes a separation degree detection unit 104, a right nostril detection unit 105, a right nostril template storage unit 106, a left nostril detection unit 107, and a left nostril template storage unit. 108, a nostril detection unit 113, a nostril template storage unit 114, and a nose detection determination unit 109.
[0094]
The degree-of-separation detection unit 104 receives an image from the image input unit 103, extracts facial feature point candidates using a degree-of-separation filter, and extracts feature point candidate information and images from the right nostril detection unit 105 and the left nostril detection unit 107. And sent to both nostril detector 113.
[0095]
The nostril detection unit 113 receives feature point candidate information and an image from the separation degree detection unit 104, receives both nostril templates from the nostril template storage unit 114, and both nostrils are present in the image based on the information. A search is made to see if both nostrils are present. Then, both nostril detection result information is sent to the nose detection determination unit 109.
[0096]
Both nostril template storage units 114 store both nostril templates in advance, and send both nostril templates to both nostril detection unit 113 as necessary.
[0097]
Here, “both nostril template” refers to data used in image matching by template matching, which is obtained by converting an image of a specific region including the right nostril and left nostril as it is or by performing principal component analysis. is there.
[0098]
The nose detection determination unit 109 receives right nostril detection result information from the right nostril detection unit 105, receives left nostril detection result information from the left nostril detection unit 107, receives both nostril detection result information from both nostril detection units 113, and This information is comprehensively determined to determine whether or not the nose is detected in the image, and the determination result and the detected nose and nostril position information are sent to the face detection unit 110.
[0099]
(2) Face Orientation Calculation Processing Procedure FIG. 7 is a flowchart showing the processing procedure of the face orientation calculation system 101 according to the embodiment. The same parts as those in FIG. Omitted and only the differences are described here.
[0100]
In step ST111, it is determined whether geometrical restrictions are imposed on whether or not both nostrils can be associated with all pairs of separation peaks.
[0101]
Next, collation with both nostril templates is performed for the peak pair whose determination result is OK (ST112). Step ST103, step ST104, and “step ST111 and step ST112” can be processed simultaneously.
[0102]
【The invention's effect】
As described above in detail, according to the present invention, it is possible to provide a robust and highly accurate face detection system even when the face direction of a person greatly changes from side to side or when wearing glasses.
[Brief description of the drawings]
FIG. 1 is a configuration diagram showing an example of a face orientation calculation system according to a first embodiment of the present invention.
FIG. 2 is a conceptual diagram illustrating both nostril templates according to the embodiment.
FIG. 3 is a conceptual diagram illustrating a single nostril template according to the embodiment.
FIG. 4 is a conceptual diagram illustrating the relationship between the degree of lateral orientation of the face and the nostril detectable range according to the embodiment;
FIG. 5 is a flowchart showing an example of face orientation calculation processing according to the embodiment;
FIG. 6 is a configuration diagram showing an example of a face orientation calculation system according to a second embodiment of the present invention.
FIG. 7 is a flowchart showing an example of face orientation calculation processing according to the embodiment;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1,101 Face orientation calculation system 2,102 Nose detection part 3,103 Image input part 4,104 Separation degree detection part 5,105 Right nostril detection part 6,106 Right nostril template storage part 7,107 Left nostril detection part 8, 108 left nostril template storage unit 9, 109 nose detection determination unit 10, 110 face detection unit 11, 111 face direction template storage unit 12, 112 face direction calculation unit 21, 31 face direction template corresponding region 22 both nostril template corresponding region 23, 33 Right nostril 24, 34 Left nostril 32 Right nostril template corresponding region 113 Nostril detection unit 114 Nostril template storage unit

Claims (6)

A face detection system for detecting a human face in an image,
Image acquisition means for acquiring the image;
A degree-of-separation detection means for extracting feature point candidates from the acquired image using a degree-of-separation filter;
A face orientation template storage means for storing a plurality of face orientation templates;
A right nostril template storage means storing a right nostril template;
Left nostril template storage means storing the left nostril template;
A right nostril detecting means for performing right nostril detected by performing template matching by matching the right nostril template the stored in the extracted feature point candidates,
A left nostril detecting means for performing left nostril detected by performing template matching by matching the left nostril template the stored in the extracted feature point candidates,
When the right nostril is detected, the front-facing and left-facing face orientation templates are selected, or when the left nostril is detected, the front-facing and right-facing face orientation templates are selected, and the selected face-facing template is acquired. Face detection means for performing face matching by performing template matching against the image that has been made ,
A face detection system characterized by comprising: Both nostril template storage means storing both nostril templates;
And both nostrils detecting means for performing both nostrils detected by performing template matching by matching both nostrils template the stored in the extracted feature point candidates,
With
The face detection means includes
The face detection is performed by selecting a face-facing template when both the nostrils are detected , matching the selected face-facing template with the acquired image , and performing template matching. The face detection system according to 1. The face detection means includes
A face search range is limited to a predetermined range based on the detected right nostril or left nostril, and the face direction template is moved within the limited face search range to perform face detection. The face detection system according to claim 1. The face detection means includes
The face search range is limited to a predetermined range based on the detected nostril, right nostril, or left nostril, and the face direction template is moved within the limited face search range to perform face detection. The face detection system according to claim 2. A face detection method for detecting a human face in an image,
An image acquisition step of acquiring the image;
A separation degree detection step of extracting feature point candidates from the acquired image by a separation degree filter;
A right nostril detecting step of performing right nostril detected by performing template matching by matching the right nostril template stored in advance in the extracted feature point candidates,
A left nostril detecting step of performing the left nostril detected by performing template matching by matching the left nostril template stored in advance in the extracted feature point candidates,
When the right nostril is detected, the front-facing and left-facing face orientation templates are selected, or when the left nostril is detected, the front-facing and right-facing face orientation templates are selected, and the selected face-facing template is acquired. A face detection step of performing face detection by performing template matching against the image that has been performed ,
A face detection method comprising: A program for realizing a face detection method for detecting a human face in an image by a computer,
An image acquisition function for acquiring the image;
A separability detection function for extracting feature point candidates from the acquired image by a separability filter;
A right nostril detecting function for right nostril detected by performing template matching by matching the right nostril template stored in advance in the extracted feature point candidates,
A left nostril detection function for left nostril detected by performing template matching by matching the left nostril template stored in advance in the extracted feature point candidates,
When the right nostril is detected, the front-facing and left-facing face orientation templates are selected, or when the left nostril is detected, the front-facing and right-facing face orientation templates are selected, and the selected face-facing template is acquired. A face detection function that performs face matching by performing template matching against the image
A program for a face detection method characterized by realizing the above.

Images