JP2007026073A - Face posture detection system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face posture detection system for detecting the spatial positions of pupils and nostrils by using an illumination means relevant to a pair of separately arranged cameras, and for showing the position of the face and the direction of the face from the positional relation. <P>SOLUTION: This face posture detection system is configured to calculate the right normal vectors of a triangle plane constituted of two pupils and a right nostril, and to calculate the left normal vectors of the triangle plane constituted of the two pupils and the left nostril in the same way, and to calculate the composite vectors of the right and the left normal vectors, and to arrange in advance the head of a testee at a fixed calibration position determined in advance for the pair of cameras, and to record pairs of distances between the two pupils and two nostrils on a two-dimensional image and the rates of the distances in a calibration step which is used for the position estimation of the nostrils. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a face posture detection system capable of acquiring a three-dimensional position of a pupil and a nostril and obtaining a face direction vector that can represent the face direction based on the three-dimensional position.

Face orientation detection is important for humans to measure the direction of attention, and is thought to be useful for preventing unnecessary driving by automobile drivers. Many applications (Patent Documents 1 to 7) and research have been made. ing.
In addition, for the same purpose, the inventor of the present invention specifies a coordinate position of the pupil and nostril by stereo measurement, and based on them, detects the position and direction of the head (Patent Document 8) and pupil In order to facilitate the detection of the pupil, the invention of a real-time pupil position detection system (Patent Document 9) that obtains the data of the bright pupil and the dark pupil and detects the two-dimensional coordinates of the pupil with high accuracy is disclosed.
JP 05-215531 A JP 09-251342 A Republished WO02 / 007095 JP 2004-94491 A JP 2004-234367 A JP-A-8-106519 JP 2000-210270 Japanese Patent Application No. 2004-73998 Japanese Patent Application No. 2004-170506

Until now, the approximate luminance distribution of the entire face is detected with a video camera, and the direction of the face is estimated from the change, or an ellipse is applied to the shape of the entire face and the direction of the face is detected from the flatness of the ellipse. There is a way. However, accuracy cannot be expected. In addition, a method for capturing facial features (for example, the eyes, eyes, nostrils, both ends of the lips) with a camera and estimating the face direction from their positional relationship, There is a method of measuring the three-dimensional positions of the above-described facial feature points from the principle of triangulation with a camera and capturing the face direction from the positional relationship (Non-Patent Document 1).
However, it is troublesome in that a template must be prepared in advance in order to search for an image of facial feature points by the template matching method. Furthermore, it is vulnerable to environmental light fluctuations. As a method that can solve the problem relatively, there is a method of detecting a black eye and a nostril (center of two nostrils) that are considered to be most easily detected as facial feature points, and detecting a face direction from the positional relationship between them ( Non-patent document 2).
This method can achieve a certain degree of accuracy and robustness if the lighting conditions are good, and at the same time, it takes advantage of the fact that black eyes and nostrils have a nearly circular shape, so no template image is required to detect them. By including these plausible positional relationships (for example, there are nostrils below the black eyes; the black eyes are far apart) in the detection condition, there is no need for learning. However, the detection accuracy and stability of black eyes are not necessarily high, and it is difficult to obtain accuracy.
Yoshio Matsumoto and A. Zelinsky, "Y.Matsumoto and A. Zelinsky. An Algorithm for Realtime Stereo Vision Implementation of Head Pose and Gaze Direction Measurement.In Proc. Of Fourth Int. Conf. on Automatic Face and Gesture Recognition (FG'2000), pages 499-505, 2000) Kentaro Hayashi, Manabu Hashimoto, Kazuhiko Kusumi “Estimation of Face Direction by Tracking Facial Feature Points with Robustness and Precision” IEICE, Vol.J84-D2 No.8, pp.1762-1771

The inventor has applied for the invention described in Patent Documents 8 and 9 described above as an invention relating to the human head. The invention of “Method for detecting head direction using pupil and nostril” (Patent Document 8) already filed solves the problem caused by the orientation of the face. In the invention described in Patent Document 8, the three-dimensional coordinates of these three points are obtained by triangulation from the coordinates of the intermediate positions of the two nostrils and the coordinates of the two pupil positions in each camera image. A normal vector of a plane passing through these three points was obtained and used as the face direction.
However, in the above method, if the face direction exceeds ± 15 degrees to the left and right, either nostril tends to be undetectable.

According to the “real-time pupil position detection system” of the present inventor's application (Patent Document 9), the position of the pupil can be detected extremely stably.
Prepare at least two cameras and install them as stereo cameras so that triangulation can be performed. Near-infrared light sources (LEDs) are arranged on the optical axis of each of these cameras or around the opening. These two sets of light sources are alternately flashed in synchronization with the camera field signal. In each camera, in a field where a light source attached to the camera is lit, an image (bright pupil image) in which the pupil is bright can be obtained. This is because the light source is close to the opening of the camera and corresponds to a so-called red-eye phenomenon. On the other hand, the pupil appears dark in the field where the light sources opposite to each other are lit (dark pupil image). Therefore, in each camera, a bright pupil image and a dark pupil image are obtained alternately. By subtracting them, only the pupil is embossed, and the pupil center with high robustness can be detected by image processing.

However, compared to the above-described pupil detection, detection of the nostrils is not easy, and it is difficult to detect one of the nostrils, or there is a problem that both nostrils cannot be identified. In order to solve the above problem, the present invention improves the detection of the nostril so that the center of both nostrils can be obtained.
Then, based on the obtained nostril position, the right normal vector from the center of gravity of the triangle in the plane of the two pupils and the right nostril is calculated. Similarly, the triangle of the triangle in the plane of the two pupils and the left nostril is calculated. The left normal vector from the center of gravity is calculated.
An attempt is made to obtain a face direction vector by combining these normal vectors.
That is, the object of the present invention is to detect the face posture that can better represent the subject's direction based on the data of both pupils and both nostrils, and can detect the vector (having robustness) reliably. To provide a system.

In order to achieve the object, a face posture detection system according to claim 1 according to the present invention is provided.
A face posture detection system that detects a spatial position of a pupil and a nostril using a lighting unit associated with a pair of cameras that are spaced apart from each other, and indicates a face position and a face direction based on the positional relationship between them. ,
Measuring the spatial coordinates corresponding to the three-dimensional position of the pupil and nostril corresponding to the center points of the two pupils and the two nostrils, and acquiring the nostril and pupil position by measuring or estimating the pupil position;
A right normal vector calculating step of calculating a normal vector of a triangular plane composed of two pupils and a right nostril;
A left normal vector calculating step of calculating a normal vector of a triangular plane composed of two pupils and a left nostril;
A combined vector calculating step for calculating a combined vector of the left and right normal vectors.

The face posture detection system according to claim 2 of the present invention is the face posture detection system according to claim 1,
A distance between a pair formed between two pupils and two nostrils on a two-dimensional image by placing a subject's head in a predetermined calibration position predetermined with respect to the pair of cameras. A calibration step to record the ratio between distances; and
When any nostril of the subject cannot be detected, a step of estimating a nostril that could not be detected based on the calibration data is provided.

The face posture detection system according to claim 3 of the present invention is the face posture detection system according to claim 1 or 2,
The positional relationship between the pair of cameras, the associated illumination means, and the subject is:
The illumination means is a position where the nostril is irradiated obliquely downward when the subject is in a normal posture or the calibration position.
The face posture detection system according to claim 4 of the present invention is the face posture detection system according to claim 1 or 2,
The illumination means is an illumination means that enables dark illumination and bright illumination to the pupil.

The face posture detection system according to claim 5 of the present invention is the face posture detection system according to claim 2,
Once the left and right nostrils are detected, use the prediction model to set up a window at the position where the nostrils are predicted to exist, and even if one nostril cannot be detected, the detected nostrils are left and right, A step for identifying either one is provided.
The face posture detection system according to claim 6 of the present invention is the face posture detection system according to claim 1,
The step of obtaining the center of gravity of the center of gravity of each triangular plane is further provided, and the direction of the composite vector is the face direction when the center of gravity of the center of gravity is used as the face position.

Three-dimensional positions of distinct feature points of the pupil and both nostrils can be detected, and the face direction vector can be detected using this.
Even if one nostril cannot be detected, it is possible to estimate the position and detect the face direction vector based on both pupils and both nostrils, so the face direction is highly robust without complicating the system. Vector detection is possible. A point representing the position of the face can be detected in the same manner, and the vector can be the direction at that point. It has been proved by experiments that a wide range of face direction detection of ± 45 degrees in the horizontal direction and correct detection in the vertical direction of ± 15 degrees or more in the vertical direction are possible.

Embodiments of a face posture detection system according to the present invention will be described below with reference to the drawings. FIG. 1 is a schematic plan view showing the arrangement of a camera and a subject in a system according to the present invention, and shows a perspective view related to FIG. FIG. 2 shows a state in which the nostril center is photographed when the subject or the user turns his face and eyes obliquely in the horizontal direction. The pupil is a dark pupil.
Illuminating means 5 and 6 are arranged so that the pair of cameras 1 and 2 arranged apart from each other irradiate the subject or the user obliquely from below (see FIG. 20).
The illumination means 5 and 6 are configured by arranging near-infrared light sources (LEDs) on the optical axis of each camera or around the opening. These two sets of light sources are alternately flashed in synchronization with the camera field signal. In each camera, in a field where a light source attached to the camera is lit, an image (bright pupil image) in which the pupil is bright can be obtained. This is because the light source is close to the opening of the camera and corresponds to a so-called red-eye phenomenon. On the other hand, the pupil appears dark in the field where the light sources opposite to each other are lit (dark pupil image). Therefore, in each camera, a bright pupil image and a dark pupil image are obtained alternately. By subtracting them, only the pupil is embossed, and the pupil center with high robustness can be detected by image processing.

As described above, the position of the camera is lowered so that the nostrils can be easily seen. For example, about 20 to 30 degrees is appropriate. A wide camera is placed below the display.
The distance between the cameras 1 and 2 is 100 mm, and the multiple light source LEDs attached to the front of the camera are alternately blinked in field units.
In this embodiment, in order to facilitate detection of the position of the pupil, bright hole illumination and dark hole illumination are provided to each camera 1 and 2 in the relationship between each camera and the head 10 of the subject (user). Illuminating means 5, 6 and the like arranged so as to maintain a positional relationship are prepared. The camera and the illuminating means are operated in conjunction to photograph the head of the subject, and images of each pupil and nostril are acquired by bright hole illumination and dark hole illumination. The bright hole illumination and the dark hole illumination are not essential constituent requirements.
It is assumed that the subject (user) sits at a location about 50 cm to 80 cm from the camera and the subject (user) faces his face in the direction in which the camera exists. These arrangements are the same as in the case of the cited reference 8 except for the position of the illumination means.

  The face posture detection system according to the present invention basically obtains data of two acquired pupils and two nostrils using the above-described apparatus. This data includes a case where data of one of the nostrils is estimated in a step described later. The other difference from Patent Document 8 is that the face direction vector detection step is based on a single triangle in the case of Patent Document 8, whereas in the case of the present invention, a triangular plane composed of two pupils and a right nostril is used. Calculate the right normal vector from the center of gravity, calculate the left normal vector from the center of gravity of the plane with two pupils and the left nostril, and based on these, determine the face direction vector, It is in the point which improved robustness. Hereinafter, a description will be given centering on the case where data of one of the nostrils is estimated in the steps described later.

  There are many dark parts in the background and face other than the face as well as the nostrils, and it takes a long calculation time to detect the nostrils, or lacks robustness. Therefore, the nostril is detected by utilizing a highly robust pupil detection technique as follows. First, find the midpoint of the two pupils. A large rectangular window is set so that the center of the window coincides with the position of the nostril when facing forward (not including the pupil). The analysis is based on a dark pupil image in which the nostrils appear darker, and is not a bright pupil image or a difference image. The above-mentioned large window is set in the dark pupil image, and only the inside is set as the search target image. First, 0.8% of pixels from the lower luminance are detected by the P-tile method. The value of 0.8% is intentionally selected to be small so that portions other than the nostrils are not erroneously detected. However, with this value, the image looks like a faint nostril. Therefore, the image is repeatedly expanded and contracted (morphological processing), and an area similar to the actual nostril shape is created from the blurred image. Further, labeling is performed to select two blocks from the larger one, and in each of them, the center of the rectangle formed from the upper, lower, left and right end points, the aspect ratio and area of the rectangle are obtained. If the aspect ratio is smaller than 0.5 or larger than 0.7 and the area is smaller than 100 pixels or larger than 300 pixels, it is determined that the image is not a nostril (when the entire image is 640 × 240 pixels). If not, a small window of 30 pixels × 30 pixels is applied to the original dark pupil image centered on the center of the rectangle described above, and within each small window, 5% of pixels from the lowest luminance are obtained by the P-tile method. Extract. Similarly, the morphological process is repeated and labeling is performed to obtain the largest block. If the area of the block is 130 pixels or more and 70 pixels or less, it is determined that it is not a nostril, otherwise it is determined that it is a nostril, and the center of the rectangle formed by the top, bottom, left, and right end points is the center of the nostril Ask.

One of the reasons for detecting a nostril by providing a large window and a small window as described above is that when a nostril is detected with a single threshold value in the large window described above, two faces are detected when the face is rotated from the front. This is because the optimum threshold value for detecting each of the nostrils changes, and the optimum threshold value is given to each of the two nostrils separately.
When two nostrils are detected in a certain frame, the left and right are determined from the size of the x-coordinates of their centers. Once the nostrils are detected, the Kalman filter is used to predict the respective nostril position in the next frame, giving them a small window from the beginning in the next frame's dark pupil image, and only inside that is the above mentioned small window. Search for nostrils as in the window analysis.

  If it is determined that there are no nostrils in both of the two small windows, in the next frame, the two small windows are canceled and a large window is set again, in which the nostrils are removed as described above. To detect. If it is determined that there is no nostril in one small window, the center coordinates of the nostril determined to be absent are estimated as described above. Furthermore, while maintaining the information indicating which of the left and right nostrils is determined to have a nostril, the position of the ipsilateral nostril in the next frame is predicted by the Kalman filter, and the position is reduced to the predicted position. A window is set in advance, and a nostril search is performed in the window. (With regard to the nostril determined to be not, try to detect the nostril by setting the threshold in the same way as described above (P-tile method, 0.4%) in the large window other than the set small window. Can also.)

The basic flow of measurement will be described with reference to the drawings. FIG. 2 shows an example in which the nostril center is detected when the user turns his face and eyes obliquely in the horizontal direction.
FIG. 3 is a schematic diagram illustrating how to provide a large window. FIG. 4 shows (a) a dark pupil image and (b) a bright pupil image used for image processing.
The nostril is detected from the dark pupil image. The reason is that since the light source for obtaining the eye pupil is near the optical axis of the camera, it is easy to illuminate the back of the nostril, and when viewed from the camera, it is difficult to produce a luminance difference from places other than the nostril. Note that an image of one field is obtained with a resolution of 640H × 240V.

(Basic flow of detection of both pupils and nostrils)
Step 1. The center coordinates of both pupils are acquired.
Step 2. The average coordinate of the center coordinates of both pupils is calculated, and a large window is set at a position where the nostril does not come off even if the head rotates with reference to the average coordinate (but below the two pupils). 3). The nostril image search is performed in it.
Step 3. Get the nostril center coordinates within the large window.
Step 4. A small window of 30H × 30V is applied around the acquired nostril center coordinates.
Step 5. After the small window is applied, the position of the nostril of the next frame is predicted by the Kalman filter, and tracking is performed while the small window is applied to the nostril.
If it is determined that both the left and right nostrils are not present, the small window is canceled and steps 1 to 4 are repeated again. When one of the left and right nostrils is removed (when it is determined that it does not exist) or when only one of the left or right nostrils is detected in the large window, use the data obtained from user calibration. The nostril image center coordinates are estimated and nostril image search is performed.
The reason for providing a small window is that the nostril can be detected more reliably in order to set a threshold value therein.
A nostril image detection algorithm based on the above image processing was executed.
In this way, by applying the window to the nostril region and the left and right nostrils for tracking, the nostril can be detected stably without being affected by noise outside the window (other than the nostrils). Details of this image processing algorithm will be described later.

After detecting the pupil image, the nostril image is detected. Pupil detection by image processing is performed from a difference image between a bright pupil image and a dark pupil image.
In nostril image detection by image processing, a nostril image cannot be detected from a difference image of pupil image detection. In addition, in a bright pupil image, there may be a high-luminance part in the nostril image, which cannot be detected stably. Therefore, nostril detection is performed from a dark pupil image in which the nostril image has a lower luminance.

  In an actual image, the illumination conditions are poor and the luminance value of the background portion and the luminance value of the target portion are not constant throughout the image. Therefore, variable threshold processing is used instead of fixed threshold processing. First, a luminance distribution is created in order to binarize the remark area of the Even image. The nostril region of the large window is shown in FIG.

The area of the nostril image in the window is approximately 70 to 130 pixels, and the area of the nostril area is 9600 pixels in the large window (60 × 160) and 900 pixels in the small window (30 × 30). Thus, the inside of the window is binarized by the P tile method. In this embodiment, the area ratio is 0.8% for the large window and 5% for the small window. The luminance distribution and threshold value of the large window and the small window at this time are shown in FIGS. Also, images obtained by binarizing the large window and the small window are shown in FIGS. 10 and 11 (however, the bright part is black (LOW) and the dark part is white (HIGH)).
Although the threshold is different between the large window and the small window, the binarized image of the nostril changes between them, and it can be predicted that the nostril can be detected more reliably in the small window.

(Morphology treatment)
10 and 11 show binary images in a large window and a small window. It is difficult to detect an accurate nostril center from such a binary image. Therefore, after the morphological process is performed on the binary image, the center is obtained. The processes used here are an expansion process and a reduction process. Here, the dilation process is a process of converting an image of interest into a HIGH pixel when at least one HIGH pixel is present in the vicinity of the image of interest in the binary image, and the reduction process is noted in the binary image. This is processing for converting a pixel of interest into a LOW pixel when there is at least one LOW pixel in the vicinity of 8 pixels.
First, reduction processing is performed after expansion. This process is called a closing process. 12 to 14 show a process of performing the above processing in a large window.
FIG. 12 shows an image after the closing process. Next, expansion processing is performed as shown in FIG. Finally, the closing process is performed again (FIG. 14).
As described above, the center of the nostril is accurately obtained.

(Judgment method of nostril and calculation of nostril center coordinates)
First, labeling is performed in order to consider the HIGH region of the binary image separately for each set. Consider a rectangle formed from top, bottom, left and right points for each set separated by labeling. When the aspect ratio of each rectangular area is smaller than 0.5 or larger than 0.7 and each rectangular area is smaller than 100 or larger than 300, it is determined that the image is not a nostril image. If it is determined that the image is a nostril image, the center of the rectangle is obtained and used as the center coordinates of the nostril image. FIG. 15A shows an example of the result of calculation of the nostril image center in the large window, and FIG. 15B shows an example of the result of calculation of the nostril image center in the two small windows.

(Application of Kalman filter)
Using the detected nostril center coordinates, the nostril position of the next frame is estimated by the Kalman filter.
The tracking of the nose in the stereo camera improves the accuracy and processing speed of nostril detection by applying a window and analyzing the limited location. However, there is a problem that if the movement of the head is large or the movement is fast, tracking cannot be performed unless the window size is increased. Therefore, by using a Kalman filter so that tracking can be performed even in a small window size, the position change of the nostril is predicted and tracking is performed, so that stable nostril detection can be performed even when the head moves.

(User calibration)
Next, user calibration will be described with reference to FIG.
The pupil position, nostril position, and nostril shape differ depending on the user. In addition, since the direction considered to be the front also varies depending on the user in the face direction, it is necessary to grasp in advance the user's pupil position, nostril position, nostril shape, and the face direction considered by the user himself / herself (hereinafter referred to as user calibration). is there. As shown in FIG. 16, for example, a circle having a diameter of 1 cm is displayed at the center of the display. Next, gaze at this circle (turn the line of sight), and face the user for 5 seconds in the direction that the user considers to be the front. During this time, the nostril detection process and the face direction calculation process are performed, and the face direction that the user considers to be the front is the front direction of the user. User calibration is to obtain two pupil center coordinates and two nostril center coordinates in the image of each camera at this time. These pieces of information are used when one nostril cannot be detected, and are unnecessary when both nostrils can be detected.

(Nostril estimation method)
When the user rotates his / her head, it is difficult to always detect both right and left nostril images. Therefore, when only one nostril image (hereinafter referred to as one nostril) is detected, a nostril image not detected is estimated.
When detecting the nostril from the nostril region, if the nostril area is larger than 130 or smaller than 70, it is excluded from the nostril object. This is because the nostrils have no distinctive features, and noise may be erroneously detected as nostril images. Therefore, when only one nostril is detected, the center of another nostril is obtained.
The right pupil center obtained from the user calibration (X _RP0, Y _RP0), left pupil center (X _LP0, Y _LP0), right nostril center (X _RN0, Y _RN0), left nostril center (X _LN0, Y _LN0) (FIG. 17A). Next, the right pupil center at a certain point in time when the face is moved is (X _RP , Y _RP ), left pupil center (X _LP , Y _LP ), right nostril center (X _RN , Y _RN ), left nostril center (X _LN , Y _LN ) (FIG. 17B). FIG. 17B shows a case where the left nostril is not detected.

The interpupillary distance D _P0 and the nostril distance D _N0 of the calibration are obtained.

D _P0 = {(X _RP0 −X _LP0 ) ² + (Y _RP0 −Y _LP0 ) ² } ^1/2 (1)

_{D N0 = {(X RN0 -X} LN0) 2 + (Y RN0 -Y LN0) 2} 1/2 (2)

Next, the pupil inclination I _P and the inter-pupil distance D _P at a certain point in time when the face is moved are obtained.

I _P = (Y _RP −Y _LP ) / (X _RP −X _LP ) (3)

D _P = {(X _RP −X _LP ) ² + (Y _RP −Y _LP ) ² } ^1/2 (4)

Both pupils and nostrils are always considered parallel. At this time, the nostril inclination I _N is as shown in Equation (6).
I _N = I _P (6)

The internostral distance at a certain point of time when the face is moved is obtained from the interpupillary distance obtained by user calibration, the internostral distance, and the interpupillary distance at a certain point when the face is moved.
D _P : D _N = D _P0 : D _N0

D _N = (D _N0 / D _P0 ) × D _P (5)

I _N and D _N are as shown in equations (6) and (7).

I _N = (Y _RN −Y _LN ) / (X _RN −X _LN ) (6)

D _N = {(X _RN −X _LN ) ² + (Y _RN −Y _LN ) ² } ^1/2 (7)

The center coordinate of the left nostril can be expressed as:

X _LN = X _RN − {D _N ² / (1 + I _N ² )} ^1/2 (8)

Y _LN = Y _RN − {(D _N ² · I _N ² ) / (1 + I _N ² )} ^1/2 (9)

However, if the left nostril could not be detected, (X _LN , Y _LN ) is unknown, so I _N and D _N cannot be obtained from Eqs. (6) and (7). Alternatively, to calculate the I _N (3) and (6), (3) calculating a D _N from equation (5), by substituting them (8) and (9) , The center coordinates (X _LN , Y _LN ) of the left nostril are estimated.
Conversely, if the right nostril cannot be detected, (X _RN , Y _RN ) is obtained by the following two equations.

X _RN = X _LN + {D _N ² / (1 + I _N ² )} ^1/2 (10)

Y _RN = Y _LN + {(D _N ² · I _N ² ) / (1 + I _N ² )} ^1/2 (11)

は、次の（１２）式で与えられる。
ｎ_LX＝（Ｙ_NL−Ｙ_PL）（Ｚ_PR−Ｚ_PL）−（Ｙ_PR−Ｙ_PL）（Ｚ_NL−Ｚ_PL）
ｎ_LY＝（Ｚ_NL−Ｚ_PL）（Ｘ_PR−Ｘ_PL）−（Ｚ_PR−Ｚ_PL）（Ｘ_NL−Ｘ_PL） （１２）
ｎ_LZ＝（Ｘ_NL−Ｘ_PL）（Ｙ_PR−Ｙ_PL）−（Ｘ_PR−Ｘ_PL）（Ｙ_NL−Ｙ_PL）

同様に、左瞳孔中心と右瞳孔中心と右鼻孔中心を通る平面の法線ベクトル

が求められる。これらの合成ベクトルを、最終的な顔方向ベクトルとして

によって求める。 (Face direction calculation method)
The three-dimensional coordinates of the four points of the left pupil center, right pupil center, left nostril center, and right nostril center are obtained. Now, the three-dimensional coordinates of the center of the right pupil, the center of the left pupil, the center of the left nostril, and the center of the right nostril are respectively (X _PR , Y _PR , Z _PR ), (X _PL , Y _PL , Z _PL ), (X _NL , Y _NL , Z _NL ) and (X _NR , Y _NR , Z _NR ) are calculated.
Normals of the plane passing through the left pupil center, right pupil center, and left nostril center

Is given by the following equation (12).
_{n LX = (Y NL -Y PL} ) (Z PR -Z PL) - (Y PR -Y PL) (Z NL -Z PL)
_{n LY = (Z NL -Z PL} ) (X PR -X PL) - (Z PR -Z PL) (X NL -X PL) (12)
n _LZ = (X _NL −X _PL ) (Y _PR −Y _PL ) − (X _PR −X _PL ) (Y _NL −Y _PL )

Similarly, the normal vector of the plane passing through the left pupil center, right pupil center, and right nostril center

Is required. These combined vectors are used as the final face direction vector.

Ask for.

すなわち、前記ベクトルは空間内の一点（Ｘ_C,Ｙ_C,Ｚ_C ）における方向を示すことになり、顔の位置と顔の方向を示すものとなる。
なお、顔の位置（Ｘ_C,Ｙ_C,Ｚ_C ）として、次式のように２個の瞳孔位置と２個の鼻孔位置の重心を用いても良いことは自明である。
（Ｘ_C,Ｙ_C,Ｚ_C ）
＝｛（Ｘ_PR＋Ｘ_PL＋Ｘ_NR＋Ｘ_NL）／４，（Ｙ_PR＋Ｙ_PL＋Ｙ_NR＋Ｙ_NL）／４，
（Ｚ_PR＋Ｚ_PL＋Ｚ_NR＋Ｚ_NL）／４｝ （１８） FIG. 18 shows the relationship between the normal vector of each plane and the face direction vector.
The horizontal angle α _H and the vertical angle β _V of the face are
α _H = −tan ⁻¹ (n _X / n _Z ) (13)
β _V = −tan ⁻¹ {n _Y / (n _X ² + n _Z ² ) ^1/2 } (14)
Given in.

FIG. 19 shows the correspondence between the vector shown in FIG. 18 and the photograph of the subject.
Further, the position of the face can be obtained as follows. In FIG. 18, the centroids (X _R, Y _R, Z _R ) and (X _L, Y _L, Z _L ) of the two triangles are given by the following equations, respectively.

(X _R, Y _R, Z _R )
= {( _XPR + _XPL + _XNR ) / 3, ( _YPR + _YPL + _YNR ) / 3, ( _ZPR + _ZPL + _ZNR ) / 3} (15)

_{(X L, Y L, Z} L)
= {( _XPR + _XPL + _XNL ) / 3, ( _YPR + _YPL + _YNL ) / 3, ( _ZPR + _ZPL + _ZNL ) / 3} (16)

Further, these centroids (X _C, Y _C, Z _C ) are given by the following equations.

(X _C, Y _C, Z _C )
= {(X _R + X _L ) / 2, (Y _R + Y _L ) / 2, (Z _R + Z _L ) / 2}
(17)

Finally, the face posture that combines the face position and face direction information passes through (X _C, Y _C, Z _C ),

That is, the vector indicates the direction at one point (X _C, Y _C, Z _C ) in the space, and indicates the face position and the face direction.
It is obvious that the center of gravity of two pupil positions and two nostril positions may be used as the face position (X _C, Y _C, Z _C ) as in the following equation.
(X _C, Y _C, Z _C )
= {( _XPR + _XPL + _XNR + _XNL ) / 4, ( _YPR + _YPL + _YNR + _YNL ) / 4,
(Z _PR + Z _PL + Z _NR + Z _NL ) / 4} (18)

(Nostril detection rate measurement experiment)
After performing the user calibration, the subject was allowed to sit in a place about 60 cm from the front of the camera. As shown in FIG. 20, the test subject has a marker attached at a position corresponding to a vertical angle of 0 degrees and a horizontal angle of 0 degrees (front), ± 15 degrees, ± 30 degrees, and ± 45 degrees. The face was turned in the direction of, and 400 frames were taken. During that time, subjects were instructed not to move their heads.
In this experiment, three subjects were tested. Table 1 shows the success, failure, blinking, the number of times of nostril estimation and the success rate (success / (success + failure)) of 400 frames of nostril detection for image processing of one camera. Here, in the case of blinking, since the face direction cannot be obtained, it is excluded when obtaining the success rate.

The nostril shapes of the three subjects were different, but as can be seen from Table 1, the nostril image detection rate was 97.9% or more for subjects A, B, and C in the horizontal head rotation from −45 ° to 45 °. (Average: 99.5%).
As the rotation angle was changed from −30 degrees to −45 degrees, or from 30 degrees to 45 degrees, the nostril estimation increased in all subjects. It is considered that the nostril image was successfully detected because the nostril center estimated by the single nostril estimation was almost the same position as the nostril center in the actual image.
Further, when detecting the face direction, it is necessary to detect four points of both pupils and both nostrils. Therefore, it was considered that the number of frames when the subject blinked was not affected by the success rate of nostril detection. The number of blinking frames for each subject at one time was 4 to 5 frames in the present pupil detection system.
From this result, it has been shown that nostril image detection is not a problem even if the nose shape or nostril shape changes, and that nostril image detection is possible in the range of −45 ° to 45 ° even in the horizontal rotation motion.

(Face direction detection experiment)
FIG. 21A shows that after performing user calibration, the subject sits in a place about 60 cm from the front of the camera, and the face direction is moved from the center (0 degree) to 45 degrees to the right and returned to 0 degree. The estimated value of the horizontal face direction is shown. FIG. 21B shows an estimated value of the vertical face direction when the head is moved within a range of ± 15 degrees up and down around the front.
As can be seen from the figure, the face direction of each subject could be detected although there was some variation in the rotation angle of the face direction in the horizontal and vertical directions to each index conducted in this experiment.
Moreover, although the rotational movement of the head was performed to each index, it was below or above the target rotation angle. It is also possible that the angle at which each subject thought that the subject had rotated to the index was different from the target angle.

  The face posture detection system according to the present invention can reliably calculate the face direction vector representing the face direction within a considerable range, so that it can detect the driving state of the driver of the driving support system such as an automobile and others. It can be widely used for detecting the user's pointing direction.

It is a top view which shows arrangement | positioning of a camera and a test subject with the system by this invention. It is a figure which shows the state by which the nostril center is image | photographed when a user has turned his face and eyes to the diagonal direction. The pupil is a dark pupil. It is the schematic for demonstrating the provision state of a large window. It is a figure which shows the (a) dark pupil image and (b) bright pupil image which are used for an image process. It is a figure which shows the state which applied the large window to the dark pupil image. It is a figure which shows the state which applied the small window to the dark pupil image. It is a graph which shows the brightness | luminance histogram in the large window of a dark pupil image. It is a graph which shows the brightness | luminance histogram in the small window given to the left nostril in the dark pupil image. It is a graph which shows the brightness | luminance histogram in the small window given to the right nostril in the dark pupil image. It is the image after binarizing the inside of the large window. It is the binarized image in two small windows, the left shows the binarized image of the left nostril, and the right shows the binarized image of the right nostril. The image after a closing process is shown. The image after an expansion process is shown. The image after performing the closing process again is shown. It is a figure which shows the state which calculated the nostril image center in a large window. It is a figure which shows the state which calculated the nostril image center in a small window. It is the schematic for demonstrating the principle of user calibration. (A) is a schematic diagram for explaining the positional relationship between the pupil and the nostril during user calibration, and (b) is a schematic diagram for explaining the positional relationship between the pupil and the nostril when the face is rotated. It is the schematic which shows each normal vector and face direction vector (synthesis | combination vector) of the plane containing a left-right pupil and a left nostril, and the plane containing a left-right pupil and a right nostril. It is the schematic which shows a response | compatibility with the vector shown in FIG. 18, and a test subject's photograph. It is the schematic which shows the direction of the target in a nostril detection rate measurement experiment. It is a graph which shows the change of the rotation angle of the horizontal direction in the face direction detection experiment which rotated the head in the horizontal direction. It is a graph which shows transition of the rotation angle of the perpendicular direction in the face direction detection experiment which rotated the head in the perpendicular direction.

Explanation of symbols

1, 2 Camera 3, 4 Camera opening 5, 6 Illumination means 10 Head of subject (user)

Claims (6)

A face posture detection system that detects a spatial position of a pupil and a nostril using a lighting unit associated with a pair of cameras that are spaced apart from each other, and indicates a face position and a face direction based on the positional relationship between them. ,
Measuring the spatial coordinates corresponding to the three-dimensional position of the pupil and nostril corresponding to the center points of the two pupils and the two nostrils, and acquiring the nostril and pupil position by measuring or estimating the pupil position;
A right normal vector calculating step of calculating a normal vector of a triangular plane composed of two pupils and a right nostril;
A left normal vector calculating step of calculating a normal vector of a triangular plane composed of two pupils and a left nostril;
A face orientation detection system comprising: a combined vector calculating step for calculating a combined vector of the left and right normal vectors. The face posture detection system according to claim 1.
A distance between a pair formed between two pupils and two nostrils on a two-dimensional image by placing a subject's head in a predetermined calibration position predetermined with respect to the pair of cameras. A calibration step to record the ratio between distances; and
A face posture detection system provided with a step of estimating a nostril that could not be detected based on the calibration data when an arbitrary nostril of the subject could not be detected. The face posture detection system according to claim 1 or 2,
The positional relationship between the pair of cameras, the associated illumination means, and the subject is:
The face posture detection system in which the illuminating means sets the nostril to be irradiated from obliquely downward when the subject is in a normal posture or the calibration position. The face posture detection system according to claim 1 or 2,
The face posture detection system, wherein the illumination unit is an illumination unit that enables dark illumination and bright illumination on a pupil. The face posture detection system according to claim 2,
Once the left and right nostrils are detected, use the prediction model to set up a window at the position where the nostrils are predicted to exist, and even if one nostril cannot be detected, the detected nostrils are left and right, A face posture detection system provided with a step for identifying either one. The face posture detection system according to claim 1.
The face posture detection system further comprising the step of obtaining the center of gravity of the center of gravity of each triangular plane, wherein the direction of the combined vector is the face direction when the center of gravity of the center of gravity is the face position.

Images