JP4850768B2 - Apparatus and program for reconstructing 3D human face surface data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for accurately reconfiguring a 3D human face. <P>SOLUTION: The device 86 comprises a module 150 for estimating a horopter range and a size of the human face based on a corrected image pair and calibration parameters of a stereo camera; a module 152 for estimating, for each pixel, a disparity value within the horopter range; a module 154 for reconfiguring 3D surface data of the human face; a module 156 for eliminating noise in the 3D surface data and interpolating an invalid data point with neighboring data points; a module 158 for applying a face plane to the 3D surface data and extracting those data of the 3D surface data within a predetermined distance from the face plane. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to an apparatus and computer program for real-time 3D reconstruction of human face surface data, and in particular, stereo for 2D / 3D face detection, extraction, recognition, 3D game development, animation and broadcasting. The present invention relates to an apparatus and a method for 3D reconstruction of human face plane data from image pairs in real time.

  Recently, 3D display technology and multimedia technology have advanced worldwide, and human face data can be played back in 3D in real time for monitoring, recognition, human-computer interaction and multimedia applications. Has attracted much interest (Non-Patent Documents 1 and 2). There are also various commercial products that can be used to digitally reconstruct 3D object shapes. However, these require special sensors such as laser range scanners and are not suitable for 3D reconstruction of human faces in general applications such as face recognition in the public domain or 3D television broadcasting.

The face reconstruction approach in the known literature can be divided into two categories depending on the application of sensor data. Reconstruction from mono (single) camera image data (Non-Patent Document 3) and reconstruction from stereo image data (Non-Patent Document 1). Regarding the reconstruction technique, shape formation from shading (Non-Patent Document 4), 3D face model (Non-Patent Documents 5 and 3), position of a face and a feature as a marker or a marker for model fitting (Non-Patent Document 2) , 6) etc. are used.
M.M. Chang, P. Derma, G. L. Gimel Ferb and P.I. Recklarc, “Comparative Study of 3d Face Acquisition Technology”, Computer Analysis of Images and Patterns (CAIP), 2005, pages 740-747. (M. Chan, P. Delmas, GL Gimel'farb, and P. Leclercq, "Comparative study of 3d face acquisition techniques.," In Computer Analysis of Images and Patterns (CAIP), 2005, pp. 740-747.) T.A. A. Erdem, “A New Method for 3d Face Model Generation for Personal User Interaction”, European Signal Processing Conference Proceedings, EUSIPCO, 2005. (TA Erdem, "A new method for generating 3d face models for personalized user interactions," in Proceedings of European Signal Processing Conference EUSIPCO, 2005.) V. Brandz and T.W. Better, “Deformable Model for 3d Face Synthesis”, International Conference on Computer Graphics and Dialogue Technology, 1999. (V. Blanz and T. Vetter, "A morphable model for the synthesis of 3d faces," in International Conference on Computer Graphics and Interactive Techniques, 1999.) J. et al. J. et al. Attic, P.A. A. Griffin and A.I. N. Redrick, “Statistical Approach from Shading to Shape: Reconstructing 3D Face Surface from a Single 2D Image”, Neural Computation, Vol. 8, pp. 1321-1340, 1996. (JJ Atick, PA Griffin, and AN Redlich, "Statistical approach to shape from shading: reconstruction of three-dimensional face surfaces from single two-dimensional images," Neural Computation, vol. 8, pp. 1321-1340, 1996.) A. R. Chowdal, R.D. Chelapa, S.C. Krishna Multi, and T. Vuo, “3d face reconstruction from videos using generic models”, IEEE International Conference Proceedings and Exhibition on Multimedia, 2002. (AR Chowdhury, R. Chellappa, S. Krishnamurthy, and T. Vo, "3d face reconstruction from video using a generic model," in Proceedings of IEEE International Conference on Multimedia and Expo, 2002.) B. W. Han, V. Brandz, T. Better and SW Lee, “Reconstruction of faces from a few feature points”, Proceedings of the 15th International Conference on Pattern Recognition, 2000. (BW Hwang, V. Blanz, T. Vetter, and SW Lee, "Face reconstruction from a small number of feature points," in Proceedings of 15th International Conference on Pattern Recognition, 2000.)

  Conventional stereo algorithms (see Non-Patent Document 1) often fail to accurately reconstruct 3D human faces. This is because when the search domain of the disparity search algorithm is unknown and therefore wide, the location of some pixels in the image pair does not provide unambiguous information regarding stereo correspondence.

  Thus, one of the objects of the present invention is to provide an apparatus that can accurately reconstruct a 3D human face.

  Another object of the present invention is an apparatus that can accurately and robustly reconstruct a 3D person's face even if the location of some pixels in the image pair does not give unambiguous information about stereo correspondence. Is to provide.

  According to a first aspect of the present invention, an apparatus for reconstructing three-dimensional human face surface data from a stereo camera corrected image pair includes a stereo camera calibration parameter and a corrected image pair. A horopter range estimation means for estimating a human face size and a horopter range based on the image, and for each pixel of the human face in one image of the image pair, within the horopter range in the other image of the image pair 3 for reconstructing the three-dimensional surface data of the human face based on the disparity estimating means for estimating the disparity value and the corrected image pair, the disparity value, and the calibration parameter of the stereo camera Finds invalid data points with invalid disparity values and interpolates invalid data points with adjacent data points. By using a noise erasing unit for erasing noise in the three-dimensional surface data, a fitting unit for applying the face plane to the three-dimensional surface data using a predetermined fitting algorithm, Extraction means for extracting a face within a predetermined distance from the face plane as a three-dimensional human face.

  A corrected stereo image pair is provided to the device. The horopter range estimation means estimates the horopter range and the size of the human face based on the calibration parameter of the stereo camera and the corrected image pair. The disparity estimating means estimates the disparity value of each pixel of the human face in one image within the horopter range of the other image. The search is robust and fast because the search area for disparity values is limited to the horopter range. Once the disparity value is estimated, the 3D surface data of the human face is reconstructed. After noise is eliminated from the surface data, invalid data is interpolated by adjacent data points. The fitting means applies the face plane to the three-dimensional surface data. Finally, among the three-dimensional surface data, data within a predetermined distance from the face plane is extracted, and the three-dimensional human face is reconstructed.

  Thus, an apparatus is provided that can accurately reconstruct a 3D human face without requiring any special sensors.

  Preferably, the horopter range estimation means corresponds to the first specifying means for specifying the interocular pattern in one image of the image pair and the epipolar line corresponding to the interocular pattern in the other image. A second specifying unit for specifying the pattern of the image, and a verification unit for verifying whether the face candidate near the interocular pattern in one image and the corresponding pattern satisfy the verification condition, Means for repeatedly operating the first and second specifying means and the verification means until the face candidate satisfies the authentication condition, and means for defining the face area by the face candidate satisfying the verification condition .

  More preferably, the disparity estimation means determines the correlation window size for the size of the horopter range and the face area, and the pixel that produces the highest similarity measure according to the preselected similarity measure, Means for calculating the disparity value of each pixel in the human face by searching in the image of.

  The noise erasing means defines the size of the deviation window with respect to the size of the face, and obtains a standard deviation of the pixel depth value in the deviation window, and for each pixel in the face area in one image, Means for calculating local standard deviation and average of pixel depth within a deviation window of size near each pixel using the disparity and image pair calculated for each pixel of the face region in the image of And depending on whether the deviation of the value of the depth of each pixel from the average of the deviation window near each pixel is smaller than the standard deviation value of the deviation window near each pixel, Means for classifying as valid and invalid and means for interpolating the depth values of the pixels classified as invalid with the depth values of the neighboring pixels may be included.

  Noise is erased as invalid data from the 3D surface data, and the depth values of missing invalid data points are interpolated from adjacent data points, so that the location of some pixels in the image pair is unambiguous with respect to stereo correspondence Even if no specific information is given, the face of a three-dimensional person is reconstructed robustly and accurately.

  Preferably, the fitting means includes means for finding an eye line connecting the left eye and the right eye in the three-dimensional surface data, means for finding a three-dimensional face symmetry line orthogonal to the eye line, and data to be applied to the face plane. Means for selecting points. The area extends from the eyes of the face to the chin, except for the tangled area along the face symmetry line. The fitting means further includes means for fitting the face plane to the data points to be fitted using a pre-determined fitting algorithm.

  More preferably, the fitting means uses a least square error fitting to fit the face plane to the data point to be fitted, and all errors between the face plane and the three-dimensional surface data are predetermined. A means for determining whether or not it is within a threshold and a high from the three-dimensional surface in response to determining that all of the errors are not less than a predetermined threshold Means for erasing points with errors, and means for repeatedly operating the fitting means, the judging means, and the erasing means until the judging means judges that all of the errors are smaller than a predetermined threshold value. ,including.

  According to the second aspect of the present invention, when the computer program is executed on the computer, the computer program is configured to determine the size of the human face and the horopter based on the calibration parameter of the stereo camera and the corrected image pair. A horopter range estimation means for estimating the range, and a disparity for estimating a disparity value in the horopter range in the other image of the image pair for each pixel of the human face in one image of the image pair An estimation means, a three-dimensional surface reconstruction means for reconstructing the three-dimensional surface data of the human face based on the corrected image pair, the disparity value, and the calibration parameters of the stereo camera; By finding invalid data points with parity values and interpolating invalid data points with adjacent data points Among the three-dimensional surface data, a noise erasing means for erasing noise in the three-dimensional surface data, a fitting means for applying a plane of the face to the three-dimensional surface data using a predetermined fitting algorithm, It functions as an extraction means for extracting a face within a predetermined distance from the face plane as a three-dimensional human face.

[Construction]
This embodiment utilizes a 3D face surface reconstruction algorithm based on stereo vision for the acquisition of real-time 3D face surface data. In this embodiment, the search domain of the disparity algorithm is limited and the disparity search range is selected using the horopter information of the observed individual face.

  A “holopter” in the field of stereovision is defined as a 3D volume covered by the search interval of the stereo correlation algorithm. As used in this specification, the “Horopter range” is the range of the 3D volume in front of the stereo camera.

  This embodiment further limits the size of the correlation window for the search. In this embodiment, the horopter information is estimated by a fast stereo eye tracking system. The proposed algorithm further estimates and uses the face plane for fast and accurate extraction of 3D face surface data. The 3D face surface reconstructed according to this embodiment retains the visual dynamics of the face because it retains most of the visual information with natural facial expressions.

  This embodiment relates to reconstructing and extracting face data of an observed individual person in real time in 3D.

  FIG. 1 is a diagram showing an overall structure of a 3D face posture estimation system 50 according to this embodiment. Referring to FIG. 1, the 3D face posture estimation system 50 estimates a 3D human face posture from a stereo camera 60, a monitor 64, and a stream of stereo image pairs from the stereo camera 60, and determines the 3D human face posture. A real-time 3D face posture estimation device 62 for extracting a face.

  The real-time 3D face posture estimation device 62 is realized by computer hardware and software as described later, calibration software 80 for calibrating the stereo camera 60 and outputting calibration parameters of the stereo camera 60, and calibration software A calibration parameter memory 82 for storing calibration parameters output by 80 and a correction for correcting a stereo image pair from the stereo camera 60 using the calibration parameters stored in the calibration parameter memory 82 Software 84 and 3D face reconstruction to reconstruct the extracted 3D face by analyzing the corrected stereo image pair from the correction software 84 Includes a module 86, by using the 3D facial images reconstructed, head posture in 3D space, i.e. for estimating the position and orientation, the 3D head pose estimation module 88, a.

  The real-time 3D face pose estimation device 62 is further accessible from the calibration software 80 and the correction software 84 and is connected to a 3D face reconstruction module 86 and a 3D head pose estimation module 88; And an image processing unit (GPU) 90 connected to the monitor 64.

  As will be described later, the instructions of the calibration software 80 and the correction software 84 are executed by a central processing unit (CPU) of the computer to achieve a desired function. Since the CPU is connected to the bus 92, the calibration software 80 and the correction software 84 can access the 3D face reconstruction module 86, the 3D head posture estimation module 88 and the GPU 90 via the bus 92.

  FIG. 2 is a hardware block diagram of the 3D face pose estimation system 50 of this embodiment realized by a computer. With reference to FIG. 2, the 3D face posture estimation system 50 includes a computer 100, a mouse 102, a keyboard 104, and a monitor 64 connected to the computer 100 as hardware components.

  In addition to the bus 92 and the GPU 90, the computer 100 further includes a CPU 120, a read only memory (ROM) 122, a random access memory (RAM) 124, a hard disk 126, and a DVD (Digital Versatile). Disk) a digital versatile disk drive 128 for driving media 106, a video capture board 130 for receiving a stream of stereo image pairs from stereo camera 60, and a semiconductor memory drive 134 for driving semiconductor memory 108. ,including. All the components in the computer 100 are connected to the bus 92 and are accessible to each other.

Stereo camera 60 includes a left camera 60L and a right camera 60R. The images ~ L _t and ~ R _t (t indicates time) from the stereo camera 60 ("~" symbol is originally added on the character) are not corrected.

In this embodiment, the calibration software 80 is used offline to calculate the calibration parameters A ₁ and A ₂ of the stereo camera 60. Calibration is performed to correct the radial distortion, lens eccentricity, focal length, pixel aspect ratio, baseline and orientation of each of the cameras 60L and 60R. The calibration parameters are stored in the calibration parameter memory 82. In the calibration process according to this embodiment, the user presents a predetermined pattern on the stereo camera 60. Calibration software 80 utilizes the output ~L _t and to R _t of the stereo camera 60, to calculate the parameters. Software for calibration is commercially available. For example, a small vision system (SVS) distributed by SRI International can be used.

Correction software 84 is used to correct the output stereo image ~L _t and to R _t of the stereo camera 60. Here, the correction means that the corresponding epipolar lines in the left and right images are aligned at the same level. This process is shown in FIG.

  Referring to FIG. 4, it is assumed that left and right images 170L and 170R of stereo camera 60 each include a line 172L and a corresponding line 172R. Due to the distortion of the lens and the difference in the orientation of the lens, images of the same line take different positions, and apart from the parallax, the left and right images have different shapes.

By correcting these images, the corresponding lines 172L and 172R are aligned in the image row as epipolar lines 182L and 182R in the corrected left and right images 180L and 180R. Without correction, real-time 3D face reconstruction from 3D stereo images is almost impossible. The correction can be performed by a predetermined calculation. In this calculation, the calibration parameter stored in the calibration parameter memory 82 is used. Correction software is also commercially available.

The corrected images L _t and R _t are provided from the correction software 84 to the 3D face reconstruction module 86 along with the configuration parameters A ₁ and A ₂ .

  FIG. 3 shows the overall structure of the 3D face reconstruction module 86 shown in FIG. The 3D face reconstruction module 86 is realized by software executed on the computer 100.

Referring to FIG. 3, the 3D face reconstruction module 86 uses the corrected image pair L _t and R _t and the calibration parameters A ₁ and A _2, and the horopter range ht = [ horopter estimation module 150 for estimating d ₁ , d ₂ ], disparity between corresponding pixels in the left and right images, and calculating the disparity image D _t with the calibration parameters A ₁ and A ₂ and the left camera image a disparity image module 152 to output together with L _t, the disparity image module 152 3D surface reconstruction module for reconstructing a rough 3D surface data to S _t of the face from the calculated disparity image D _t by 154 And including.

  The 3D face reconstruction module 86 further eliminates outliers or noise from the rough 3D face data to St, smooths the 3D surface data, and outputs the 3D face data St. A 3D surface extraction module 158 for estimating a face plane based on the smoothed 3D face data St, and extracting a segment to Ft consisting of voxels within a predetermined distance of the 3D face from the face plane; ,including.

  As used herein, “face plane” means a surface defined in relation to a 3D face image that approximates the surface of the face to a flat surface.

  In this embodiment, the 3D face reconstruction module 86 further includes a texture mapping module 160 for mapping the skin texture to the 3D face segment to Ft output from the 3D surface extraction module 158. The 3D face image Ft to which the skin texture is mapped is given to the monitor 64 and displayed.

<Eye tracking>
The horopta estimation module 150 uses an eye tracking algorithm in estimating the horopter. Any eye tracking algorithm would be suitable for this task. In this implementation, the pattern between the eyes (where the eye location has a lower light intensity than the cheek and nose bridge) is detected and tracked by pattern matching with an updated size. To accommodate the face scale, different scale patterns are considered during detection and a scale suitable for tracking is selected accordingly.

  The algorithm computes an intermediate representation of the input image, called the “integrated image”. An integral image is obtained by summing the pixel values in a rectangle of any size that encompasses the pixels for each pixel of the image and adding the resulting sum to that pixel. Thereafter, the light / dark relationship of the eye area of the image is filtered at high speed using a six segmented rectangular (SSR) filter. For initial verification of candidates, a support vector machine (SVM) algorithm is used.

  Using epipolar constraints, use a correlation-based template matching algorithm to estimate the stereo eye position and the user's head horopter along the epipolar line in the right camera image, Search for patterns.

As used herein, “Holoputa” means a 3D volume covered by the search interval of the stereo correlation algorithm. The “holopter range” is here the range of the 3D volume in front of the stereo camera. These terms are often used in the field of stereovision and are useful for finding the 3D coordinate points of a pixel in a stereo image pair (vs. pixel) .

For example, regarding any unknown point <X, Y, Z> in the 3D world, let (x _L , y _L ) be the coordinates of the left camera image. In order to calculate the value of <X, Y, Z> from the stereo camera system, it is necessary to find the corresponding point (x _R , y _R ) of the “right” camera image using a stereo correlation algorithm.

If there is no idea about the horopter at the unknown target point <X, Y, Z>, the search interval of the right camera is x _R = [0, x _L ] (if the image is corrected, y _R = y _L ). This has two drawbacks.

  1- Since the search range is wide, the calculation cost is high.

  2- There is a high possibility of false detection of coincidence.

If the horopter at the unknown target point <X, Y, Z> is roughly known, the search interval is x _R = [x ₁ , x ₂ ] (image corrected for the range of the right camera image In this case, y _R = y _L ) can be set.

In this case, the interval [x ₁ , x ₂ ] is a subset of the interval [0, x _L ]. Thus, the computational cost is lower and the probability of a false detection of a match is greatly reduced, thus making it possible to correct the error (if any).

  The horopter can be calculated as follows. Assume that there is an object point <X, Y, 70 cm> along the optical axis of the corrected stereo camera system. Let L <125,200> be the coordinates of the “left” camera image at this point, and R <70,200> be the coordinates of the “right” camera image at this point. The 3D reconstruction based on the stereo vision of the target point <X, Y, 70 cm> is as follows.

  If L <125,200>, R <70,200> and the camera calibration matrix are known, the coordinates <X, Y, 70 cm> of the target point can be calculated in 3D.

  If the horopter information is unknown, L <125,200> from the “left” camera image coordinates of this point can be used. First, it is necessary to obtain the corresponding point R <70,200> from the right camera image. However, all that is known about the x-axis is that this is from 0 to 125. Here, it is assumed that the symmetry point is visible in any camera image.

If the horopter information is “known”, the 3D volume of the 3D space where the object is located is known. This volume is, for example, [<X, Y, 90 cm>, <X, Y, 50 cm>]. Using this projection,
For <X, Y, 90 cm>, L <x _1L , y _1L >, R <x _1R , y _1R >
Where y _1L = y _1R since the image is corrected. Therefore, the disparity _{d 1} becomes _d 1 ₌ x 1L _{-x 1R.}

Similarly,
<X, Y, 50 cm> with _respect to L _{<x2L, y2L>} , R _{<x2R, y2R>}
Where y _2L = y _2R . Therefore disparity _{d 2} becomes _d 2 ₌ x 2L _{-x 2R.} Here _{d 2} is greater than _{d 1.}

Therefore, all points between <X, Y, 50 cm> and <X, Y, 90 cm> will have a disparity between [d ₁ , d ₂ ]. Therefore, for an arbitrary point L <x _L , y _L > in the left camera image that exists in the horopter in the above example, the search interval of the right camera image is replaced with x-axis coordinate value [0, x _L ]. It is well limited by _R = [x _L −d ₂ , x _L −d ₁ ].

The horopter size can be estimated from the face size. In this embodiment, eyes are found from both the left and right images, and first, the 3D coordinates of the nostril point between the eyes are estimated. This is _defined as <X _nb , Y _nb , Z _nb >. Now you can know the approximate size of the face. Therefore, a horopter can be defined for this face. This is because horopta is related to face size. For example, in this case, the horopter range is <X, Y, [Z _nb −10 cm, Z _nb +10 cm]> for any point on the face.

  Note that 3D X and Y coordinate values have no effect on the disparity search range. Only the Z coordinate is affected.

Accordingly, as in the previous example, the disparity search range can be defined for the right camera image. That is, for any point L <x _L , y _L > in the left camera image, the search interval in the right camera image is x _R = [x _L −d ₂ , x _L −d ₁ ]. Here, y _L = y _R.

  FIG. 5 is a flowchart of a computer program for realizing the horopter estimation module 150 shown in FIG. Referring to FIG. 5, given a calibration parameter and a corrected left and right stereo image pair, the program starts at step 200 where the inter-eye pattern of the left camera image is identified.

  Following the step 200, the program further includes a step 202 for finding a corresponding pattern along the epipolar line of the right camera image, and a step 202 followed by loosely defined heuristic constraints and SVM (support vector machine). Using step 204 to verify the face, following step 204, determining whether or not the face has been authenticated and separating the execution flow of the program in two directions, and when the face is verified in step 206 Executed to generate a face ROI (Region of Interest) and a horopter range 208. If the face is not verified at step 206, control returns to step 200. Therefore, steps 200 to 206 are repeated until the face is verified.

  FIG. 6 shows the limits of the verification range. Referring to FIG. 6, it is assumed that pixel 224L of left image 220L has a pixel 224R corresponding to right image 220R. If the image 220R is placed on the image 220L, an image 220 in which the pixels 224L and 224R are shifted by the amount of the horopter should be obtained. The maximum value of this distance is limited by the horopter size and the face size. Accordingly, the pixel 224R corresponding to the pixel 224L should be found in the right image within the distance Ht from the pixel 224L. The distance between the pixels 224L and 224R in the image 220 is referred to as “disparity”.

<Reconstruction of 3D face data>
The stereo computer vision algorithm uses disparity along with camera calibration parameters to calculate 3D (X, Y, Z) coordinates of the target point. The disparity algorithm often uses a variety of techniques such as the square of the difference or the normalized correlation coefficient method to compare the left camera image template window with the window along the epipolar line of the right camera image Depends on.

  That is, the matching algorithm needs to calculate a measure of the similarity of the 2D template window A of the left image to the 2D window B of the right image. In general, A and B can be expressed as follows.

Here, γ and Γ are arbitrary gain / scaling values of the left and right cameras, respectively. _X (where _ represents an overbar added to the letter in the formula) represents an average value, and ~ X (to is given to the letter in the formula) is the authenticity of the window data Represents the shape or texture characteristics. That is, if ~ A and ~ B belong to the same surface portion of the object, they are the same under ideal conditions.

  However, under real-world conditions, disparity search algorithms may not work well due to video noise or due to lack of intrinsic true shape characteristics that ~ A and ~ B do not hold any information. is there. Video noise can occur due to different lens focus, different viewing angles, and different lighting effects of the left and right cameras.

  To reduce the number of coincidence detection failures or false detections, facial horopters are estimated on the fly from stereo eye tracking. That is, first, the coordinates of the left and right eyes in the 3D space are calculated, and the face size is estimated from the distance between the left and right eyes. Since the size of the horopter depends on the size of the face, this is also estimated. The search domain of the disparity algorithm is then limited for this face by the horopter size. The resulting disparity is further verified by a consistency check within the pixel neighborhood. The flow of the disparity search algorithm is shown in FIG.

  FIG. 7 is a software flowchart for realizing the disparity image module 152. Referring to FIG. 7, given calibration parameters, corrected left and right image pairs, face region and horopter range, the program at step 240 defines a correlation window size for the given face size and horopter range. Beginning and following step 240, following steps 242 to find disparity values within the given horopter range of the right image using various similarity measures for each pixel in the face region Determining whether all disparities have been calculated for the pixels in the face area, and branching the control flow into two steps 244. That is, if all disparities have been calculated, program execution is terminated. Otherwise, control flow returns to step 242.

  FIG. 8 is a diagram showing how disparity is calculated for the pixel 262L of the left image 260L. Referring to FIG. 8, a window 264L is defined in left image 260L. The size of the window 264L is determined by the face size and the horopter. In the right image 260R, a window 264R having the same size as that of 264L is defined on the epipolar line in the right image 260R corresponding to the pixel 262L. Pixel 262R corresponding to pixel 262L is searched by moving window 264R as indicated by double-headed arrow 266 and using various similarity measures between windows 264L and 264R. The search space is limited within the horopter range from pixel 262L.

<3D surface reconstruction>
Given a pixel pair and its disparity from the left and right images, the depth of the voxel corresponding to the pixel pair can be determined based on the calibration parameters of the stereo camera 60. Here, the depth of the voxel means a distance from the image plane of the stereo camera 60 to the voxel in the 3D space. This operation is performed by the 3D surface reconstruction module 154.

<Noise elimination and interpolation>
The noise cancellation and interpolation algorithm eliminates spurious noise from 3D data and recovers missing voxels. In this implementation, missing voxels are estimated by linear interpolation near the pixels.

  Referring to FIG. 9, the noise cancellation and interpolation module 156 compares the depth of each voxel with the average depth and the local standard deviation of the depth within a predetermined window that contains the voxels to validate the voxel. And 3D face by applying a smoothing algorithm to valid 3D face data, and a noise elimination module 280 that eliminates noise from the 3D face data output from the 3D surface reconstruction module 154 A smoothing module 282 for smoothing the data and an interpolation module 284 for estimating missing (invalid) voxels by linear interpolation of neighboring voxels.

  FIG. 10 is a flowchart of a program for realizing the noise elimination module 280. Referring to FIG. 10, the program is given a window for a given face size for local standard deviation estimation given the reconstructed initial dense surface data associated with the face area in the left image. Starting with step 300 defining the size, and following step 300, calculating 302 the z-axis (depth) average and the local standard deviation of the pixels in the window for each pixel in the face area of the left image; , Step 302 includes determining 304 whether the deviation of the z-axis value from the average is less than the standard deviation.

  The program further executes step 306 to set the voxel measurement to valid, which is executed when the answer at step 304 is YES, and is executed when the answer at step 304 is NO. Setting the measurement invalid. After steps 306 and 308, control returns to step 302. Although not explicitly shown in FIG. 10, when all voxels have been considered in step 304, control returns from this program.

  Therefore, when control returns from this program, all voxels are labeled as valid or invalid.

  This process is shown in FIG. Referring to FIG. 11, the depth of each voxel in the face area is calculated from the left image 320L and the right image 320R using the disparity of the pixels 322L and 322R and the calibration parameter 82 (330). ). The resulting dense surface data 332 with depth values is provided to the noise cancellation program. For each voxel in the dense surface data 332, the deviation of that depth from the average depth of the window containing this voxel is calculated and compared to the standard depth deviation of the window, and accordingly the depth deviation is standard. Depending on whether it is larger than the deviation, as shown in the valid / invalid voxel map 334, the voxel is labeled as valid or invalid.

  FIG. 12 is a flowchart of a program for realizing the smoothing module 282 shown in FIG. Referring to FIG. 12, the program begins at step 350, which defines a smoothed window size for the estimated face size, given the noise-removed surface data associated with the face area of the left image. Following 350, for each pixel having a valid 3D measurement in the face area, calculate its average value in the smoothing window and assign it to that pixel as a new 3D value; Subsequently, a step 354 for determining whether or not all the pixels of the face area have been processed is included. If the answer to step 354 is yes, control exits the routine. Otherwise, control returns to step 352.

  FIG. 13 is a flowchart for realizing the interpolation module 284 shown in FIG. Referring to FIG. 13, when smoothed 3D surface data related to the face area of the left image is given, the program for each pixel having an invalid 3D measurement value in the face area is the data of the neighboring pixels. Starting with step 370 of estimating the 3D measurement value by linear interpolation of step 370, step 370 is followed by step 372 of determining whether all the pixels of the face region have been processed.

  The program further includes a step 374 of executing the smoothing algorithm shown in FIG. 12 for the face region, which is executed if the answer in step 372 is YES. When step 374 is complete, control returns from this routine.

  If the answer at step 372 is no, control returns to step 370.

<3D face plane extraction>
FIG. 14 is a flowchart of a program for realizing a program for realizing 3D face data extraction used in this embodiment. Referring to FIG. 14, given dense 3D surface data (voxels) related to the face area of the left image, the program begins at step 390, where the nail-shaped areas including the nose and mouth areas are excluded. The face plane is estimated using the voxel data between the eyes and the jaw. In this embodiment, a novel 3D face plane estimation algorithm is used at step 390. Details of this step will be described later with reference to FIGS.

  Following the step 390, the program further includes a step 392 in which all voxels whose distance to the face plane is within ± δ (δ is a predetermined threshold) are extracted as face voxels.

  The program further includes a step 394 that determines whether all voxels in the face region have been processed. If all voxels have been processed, control exits this routine. Otherwise, control returns to step 392.

  FIG. 15 is a detailed flow diagram of a program that implements the novel 3D face plane estimation algorithm utilized in step 390 of FIG. The plane of the face is estimated from the 3D face data of the part between the eyes and the chin, excluding the tongue-shaped part including the mouth and nose. However, if the head is in an extreme orientation, the 3D face data may include an outlier. For this reason, a point with a certain high error is eliminated using an error histogram between the face data and the estimated surface by using an iterative least square method. Since the repetition stops in 2 or 3 times, a heavy computational burden does not occur. The face plane parameters a, b, c and d at (aX + bY + cZ + d = 0) are the remaining 3D face data and are estimated at step 390 using the least squares solution.

  Referring to FIG. 15, this routine begins at step 410 given the dense 3D surface data (voxels) associated with the facial region in the left image, where the nose and mouth between both eyes and chin. The symmetric voxel data excluding the tangled portion including the area is extracted from both sides of the symmetric line of the face orthogonal to the 3D line of sight.

  This process is shown in FIGS. Referring to FIG. 16, when a left eye position 440 and a right eye position 442 are given in the left camera image 430, a center point 446 on a line 444 connecting the eye positions 440 and 442 is defined. A number of points 460, 462, 464,... At a specific distance from line 444 are then defined in 3D space. For each of these points, a 3D vector is defined from the center point 446. By calculating the inner product of this vector and a vector parallel to line 444, it is determined which point defines the symmetry data between both eyes and jaw.

  In FIG. 16, assuming that the point 462 is symmetric data, another set of points is provisionally defined under the point 462. Another symmetric data point will be defined from this set of points. By repeating this operation, a set of symmetrical data points 480 between the center point 446 and the jaw is defined as shown in FIG. This line is a symmetric line of the face, and in this embodiment this is called “Sline”.

  Referring to FIG. 18, for each symmetric data point 480, eg, symmetric data point 500, a particular length of line 502 parallel to line 444 is selected. These lines form the frontal face area between both eyes and chin. The width of the selected area is 1.3 times the integrated eye distance in this embodiment, and the height is bounded by the jaw.

  Referring to FIG. 18, the front face area thus defined is divided into three tangential areas. The central part of these lines is excluded and the rest of these lines is a symmetrical voxel area 504 (left face: LF) between the eyes and chin, excluding the tangled area including the nose and mouth. 506 (right face: RF). The LF and RF areas are equal to 1/4 of the entire front face area, and the central area is equal to 1/2.

  These areas 504 and 506 are shown on the face image in FIG.

  Referring again to FIG. 15, the program further includes a step 412 of estimating the face plane by fitting the plane to valid voxel data in areas 504 and 506. For computational efficiency, this fit uses only voxels on sparsely sampled lines in areas 504 and 506 (see FIG. 19), rather than all voxels.

  Following the step 412, the program further evaluates all the given voxel data by applying it to the estimated face plane and invalidates the ε% of the worst fit data, followed by the step 414. And 416 for determining whether the fitting error rate is smaller than 1 cm (10 mm). If all fit errors are less than 1 cm, control exits this program. Otherwise, control returns to step 412.

  Details of the fitting algorithm are described below. This process is an iterative least squares fitting and evaluation algorithm for face plane estimation. In general, the 3D face plane obtained by fitting from stereo data tends to cause an error because the amount of visible data of the face varies when the head moves. Thus, symmetric data from both sides of the face is selected, but this data set does not include areas for individual facial features such as nose and mouth. For this reason, as shown in FIG. 19, the front 3D face data is divided into three simple regions aligned in both eyes.

  For practical reasons, sparse sampling of the data is performed (eg, 20 lines between the eyes and jaw lines). Finally, artifacts that appear to come from partial occlusion (partial hiding) or head movement are eliminated by iterative least squares fitting and removal-addition algorithms. The parameters used in this algorithm were determined empirically by examining the corresponding face images of over 40 people.

  Given a set of extracted 3D face data, the algorithm flow can be written as:

2) Calculate the integrated Euclidean distance from the 3D face surface data between the eyes.

  3) Define a frontal face data area whose center is aligned with the face symmetry point for each row. The width of the selected area is ξ (ξ = 1.3) times the integral distance, and the height is defined by the jaw.

  4) Divide the front face area into three tangential areas. The left face (LF) flank area and the right face (RF) flank area are 1/4 of the total front face area, and the central face area is 1/2 of the total front face area.

  5) Define a sparse data row parallel to the 3D line in the 3D data area and collect the same length of data from the inside to the outside in both the LF row and the RF row (this for all rows up to the jaw) Repeat).

  6) Iteration: 3D plane parameters best fit by the least square fitting algorithm (face data coordinates Xi, Yi, Zi, where i = 1, 2,... Nk, Nk is the number of samples in the kth iteration) A, b, c and d) are estimated.

8) Evaluation: If the error is less than τ (τ = 10) millimeters or the number of repetitions is greater than κ (in this embodiment, κ = 4), proceed to step 11 (END).

  9) Remove ε (ε = 5)% of face data having high error from the set.

  10) If the error distance of previously discarded data is less than τ millimeters from the newly estimated surface, add this data to the set and go to “Repetition”.

  11) END.

  This algorithm is not greatly affected by how the parameters are selected. The width parameter ξ may be greater than 0.9 but must be less than 1.5. The error threshold parameter τ may be greater than 5 millimeters but must be less than 15 millimeters. The data removal rate may be between 1% and 10%, and the maximum number of repetitions is between 2 and 10.

  FIG. 20 shows an example of repetition from steps 412 to 416. Referring to FIG. 20, in this example, the top error histogram was obtained in the first iteration. This had a long tail exceeding 10 mm. The second iteration was performed with ε% of the worst fitting data invalidated. In the second and third iterations, the second and third histograms of FIG. 20 were obtained. You can see that the “tail” is getting shorter. However, some errors still exceed 10 mm even in the third iteration.

  As can be seen from the lowest histogram in FIG. 20, after the fourth iteration, the error is within 10 mm from the plane of the face to which all errors have been applied, and the iteration ends.

  FIG. 21 shows face extraction in step 392 of FIG. Referring to FIG. 21, a face plane 520 is estimated in step 390, and all voxels having a distance within ± δ from the face plane 520 are extracted as face voxels.

  The texture mapping module 160 shown in FIG. 3 maps the texture onto the face data extracted by the 3D face extraction module 158.

  In FIG. 22, (A) and (B) show a stereo image pair, and (C) shows the disparity. The detected position between both eyes is marked in both the left camera image and the right camera image, and these are used for face horopter estimation.

  FIG. 23 shows stereo pairs in (A) and (B), and shows their extracted and reconstructed 3D face data in (C). It can be seen that outside the face horopta, the reliability of the 3D reconstruction is low.

  FIG. 24 shows 3D face images stored from different viewing angles.

<Formation of 3D face coordinate system>
An arbitrary 3D coordinate system can be described by a vector of a surface and the normal of the surface. Therefore, the goal of this embodiment after obtaining the 3D eye position and face plane is to form a 3D face coordinate system for estimating the head pose for each frame. This is performed by the 3D head posture estimation module 88 shown in FIG. 1, and details thereof are shown in FIG.

Referring to FIG. 25, the 3D head posture estimation module 88 is coupled to receive the left and right 3D eye positions e _Lt and e _Rt and the 3D face plane parameter Pt, and the 3D surface extraction modules 158 to 3D of FIG. A 3D face coordinate system forming module 540 for forming a face coordinate system and outputting a matrix FCS _t representing the face coordinate system is coupled to receive a matrix FCSt output from the 3D face coordinate system forming module 540. A 3D head posture estimation module 542 for estimating a 3D head posture matrix M.

  In this embodiment, the 3D face coordinate system is defined by the line of sight, the face plane, and the normal of the face plane.

  Face plane parameters a, b, c and d are calculated in step 390 shown in FIG. The normal of the face plane is available and is represented by the following vector:

Let E1 and E2 be the positions of the left and right eyes of the face plane in the following 3D space.

Here, {circumflex over (Z)} ("^" is added to the letter in the formula) is the recalculated Z value on the face plane when the values of Xi and Yi are given. The expression of the line of sight (Eline) in the 3D space can be defined as follows using the points E1 and E2.

Here, t is a scalar value (tεR) of a point on Eline, and → V _x = → E ₁ E ₂ (“→” is added on the letter in the formula), Defines a vector perpendicular to the normal of the face plane.

The cross product of the vector → Vz which is the normal of the face plane and the vector → Vx obtained from the 3D line of Equation 3 becomes the vector → V _y , which is perpendicular to both → V _x and → V _y. is there. These three vectors form a face coordinate system. Therefore, the x-axis and z-axis of the face coordinate system are locked to the 3D eye position and the 3D face plane, respectively. Therefore, the formation of the 3D coordinate system can be repeated for each frame.

  Both the 3D face coordinate system formation module 540 and the 3D head posture estimation module 542 are implemented by computer program routines shown in FIGS. 26 and 30, respectively.

  Referring to FIGS. 26 and 27, the computer program routine that implements the 3D face coordinate system formation module 540, given the 3D eye positions 580L and 580R at a certain time and the 3D face plane parameters, step 560. Here, a 3D eye line 582 from the left eye position 580L to the right eye position 580R and a unit vector 586 parallel to the 3D eye line 582 are calculated. In this embodiment, the starting point of the unit vector 586 is a point 584 between both eyes.

  26 and 28, the program further continues to step 560 by assigning a unit normal vector 588 of the face plane 520 to the z-axis of the face coordinate system, and 3D line of sight to the x-axis of the face coordinate system. Assigning a unit vector 586 parallel to 582.

  26 and 29, the program further calculates the cross product 590 of the z-axis vector 588 of the face coordinate system and the x-axis vector 586 following the step 562 to obtain the y-axis of the face coordinate system. Step 564 of obtaining a vector 590 is included.

  The program routine includes a final step 566 that uses the three vectors 586, 590 and 588 to form a 3D coordinate system.

<3D head posture estimation>
The transformation representing this 3D head posture between the face coordinate system and the global (reference) coordinate system is as follows.

  A rotation-only transformation between two coordinate systems can be expressed as:

  With reference to FIG. 31, generally, the face coordinate system 632 can be considered to be a shape obtained by rotating and translating the world coordinate system 630. That is, the face coordinate system 632 and the world coordinate system 630 are associated with each other by a 3 × 3 rotation matrix R (Φ) and a 3 × 1 translation vector T as indicated by a dashed arrow 634 in FIG.

Therefore, the solution of the 3D head posture matrix R (Φ) in Equation 4 is self-evident and can be expressed as:

The rotation angle for each axis that satisfies Equation 8 is given by:

This process is realized by a computer program routine. A flowchart of this routine is shown in FIG. Referring to FIG. 30, when a face coordinate system at a certain time point is given, the routine estimates a 3D head posture with respect to the reference (common) coordinate system 630 for each frame time point according to equations (9) to (11). Begin at step 610. After step 610, control exits this routine.

[Operation]
With reference to FIGS. 1 to 30, the 3D face posture estimation system 50 described above operates as follows. With particular reference to FIG. 1, calibration is first performed. In this calibration process, a predetermined pattern plate is presented to the stereo camera 60, and the calibration software 80 calculates calibration parameters. The parameters are stored in the calibration parameter memory 82.

  In operation, the system 50 first acquires a stereo image from the stereo camera 60. The captured image is corrected by the correction software 84 using the calibration parameters stored in the calibration parameter memory 82 so that the epipolar lines correspond to the rows of the image. The corrected image is provided to the 3D face reconstruction module 86.

  With particular reference to FIG. 3, the horopter estimation module 150 estimates a face horopter using a stereo image. The horopter estimation module 150 relies on a stereo eye tracking algorithm in this implementation. Stereo eye tracking allows the horopter estimation module 150 to generate face horopter information. The face horopter information ht helps to calculate the search area ht = [d1, d2] for finding the corresponding pixels in the left and right images. The disparity search algorithm uses the horopter information and the correlation template size to obtain a better disparity result.

  The horopter information, the left and right images Li and Ri, and the calculation parameter Ai are given to the disparity image module 152. In the disparity image module 152, a disparity Dt is calculated for each pixel in the image. The 3D position of a point can be reconstructed with prior art stereo reconstruction algorithms given the <x, y> coordinates and their disparity. The 3D surface is reconstructed based on the calculated data.

  Since the reconstructed 3D face surface includes several outliers, the noise cancellation and interpolation module 156 eliminates invalid voxels and interpolates the missing voxels by interpolation between adjacent voxels. The 3D surface data is further smoothed in the noise cancellation and interpolation module 156, thereby obtaining dense 3D face surface data, which is provided to the 3D surface extraction module 158.

  The 3D surface extraction module 158 determines the points between the eyes and extracts the symmetric face voxel data between the eyes and chin. As shown in step 410 of FIG. 15 and FIG. 19, the bumpy region including the nose and mouth areas is excluded from both sides of the face symmetry line. Fit the plane using the voxels in the extracted target face voxel data and exclude this worst fit data until all the remaining symmetric face voxels are within a predetermined distance, eg, 10 mm, from the face plane. By repeating, the face plane is estimated.

  Using the face plane parameters, face voxels located at a distance of ± δ from the face plane are extracted as face voxels.

  The texture mapping module 160 maps the extracted skin texture of the face voxel, and the extracted face image is generated as shown in FIGS.

Face plane parameters are further provided to the 3D head pose estimation module 88 shown in FIG. Referring to FIG. 25, the 3D face coordinate system formation module 540 first obtains a unit vector 586 parallel to the eye line as shown in FIG. 27, and a unit vector 58 perpendicular to the face plane 520 as shown in FIG. 8 is calculated, and then a unit vector 590 which is a cross product of the vectors 586 and 588 is calculated as shown in FIG. These three vectors form a 3D face coordinate system.

<Experimental setup>
Vide stereo vision hardware and SVS software are used for this implementation. Camera calibration and correction is performed automatically using the SVS library. SVS software can capture a stereo video sequence and reconstruct stereo pairs of 3D data at 30 Hz with a total image resolution of 320 × 240. However, since the region related to the 3D reconstruction is the user's face area in this experiment, the disparity search area was limited to the area around the face horopter as described above. Therefore, the reconstructed 3D data having different depths outside the face has an inaccurate 3D estimation as shown in FIG.

  3D coordinate values are calculated in the world coordinate system. In this implementation, the world coordinate system (origin) is defined as the focus of the left camera and is the right hand coordinate system.

  In order to evaluate the accuracy of the head pose estimation scheme, we compared it with measurements obtained from marker-based head pose estimation. As shown by reference numeral 650 in FIG. 32, three black markers having a radius of 5 mm are positioned on the user's forehead so as to form an L-shape that is rotated 90 degrees clockwise by 25 mm apart, A 3D coordinate of the marker position was estimated using a stereo processing algorithm. The accuracy of the marker-based head pose estimation algorithm was ± 3 degrees due to jitter in marker position detection.

  In FIG. 33, the angle estimated by the proposed algorithm is compared with the value obtained by the marker-based algorithm. The results of this embodiment are shown by thin solid lines 660, 672, and 682, and those measured with a marker-based algorithm are shown by thick dashed lines 662, 670, and 680.

  Naturally, the forehead where the marker is positioned has a different posture from that of the face reflected in the plot due to its respective offset value. In the x, y, and z-axis angle data shown in FIG. 33, the correlation coefficients were 0.87, 0.92, and 0.98, respectively. The lower correlation coefficient for rotation about the x axis compared to the results for the y and z axes is due to the eye position jitter detected by the eye tracking algorithm. However, jitter in eye tracking can be corrected by using 3D face structure information or a better eye tracker.

CONCLUSION The above-described embodiments provide a robust 3D face extraction and 3D head pose estimation scheme suitable for human-computer interface applications under real world conditions. The proposed scheme requires no model, does not require initialization, and can extract 3D faces from a single image pair to estimate 3D head pose information. It is also robust against different facial features such as facial expressions and nose shapes.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the whole structure of the 3D face attitude | position estimation system according to embodiment of this invention. It is a hardware block diagram of 3D face posture estimation system 50 of this invention realized by computer. It is a figure which shows the whole structure of the 3D face reconstruction module 86 shown in FIG. It is a figure which shows a correction process. It is a figure which shows the detailed flow of a horopter estimation algorithm. It is a figure which shows limitation of the search range for finding the pattern between both eyes in a stereo image. It is a flowchart which shows the whole process step of the proposed disparity search algorithm. It is a figure which shows how the disparity of the pixel 262L in the left image 260L is calculated. It is a figure which shows the detailed structure of the noise cancellation and interpolation module shown in FIG. It is a flowchart which shows the whole process step of a noise elimination algorithm. It is a figure which shows the process which verifies the voxel of face data. FIG. 10 is a flowchart of a program that implements the smoothing module 282 shown in FIG. 9. FIG. 10 is a flowchart for realizing the interpolation module 284 shown in FIG. 9. It is a flowchart of the program which implement | achieves 3D face data extraction used by one embodiment of this invention. FIG. 15 is a detailed flowchart of a program that implements a novel 3D face plane estimation algorithm used in step 390 of FIG. 14. It is a figure which shows the extraction process of the symmetrical voxel data between both eyes and jaws. It is a figure which shows the extraction process of the symmetrical voxel data between both eyes and jaws. It is a figure which shows the extraction process of the symmetrical voxel data between both eyes and jaws. It is a figure which shows the extracted symmetrical voxel data on a face image. It is a figure which shows the example of the repetition for excluding outlier data. It is a figure which shows the face extraction in step 392 of FIG. (A) And (B) is a figure which shows a stereo image pair, (C) is a figure which shows the disparity map. (A) and (B) are diagrams showing stereo image pairs, and (C) is a diagram showing the extracted and reconstructed 3D face data. It is a figure which shows the 3D face image preserve | saved from the different viewing angle. 4 is a detailed block diagram of a 3D head posture estimation module 88. FIG. It is a flowchart for implement | achieving 3D face coordinate system formation module 540. FIG. It is a figure which shows the process which forms 3D face coordinate system. It is a figure which shows the process which forms 3D face coordinate system. It is a figure which shows the process which forms 3D face coordinate system. It is a flowchart which shows the computer program routine for estimating 3D head posture with respect to a reference coordinate system. It is a figure which shows how the face coordinate system 632 and the world coordinate system 630 are linked | related by 3 * 3 rotation matrix R ((PHI)) and 3 * 1 translation vector T. FIG. It is a figure which shows the setting for the measured value obtained from the marker-based head posture estimation. It is a figure which compares and compares the angle estimated from embodiment of this invention with the value obtained from the marker-based algorithm.

Explanation of symbols

50 3D face posture estimation system 60 stereo camera 62 real-time 3D face posture estimation device 80 calibration software 82 calibration parameter memory 84 correction software 86 3D face reconstruction module 88 3D head posture estimation module 130 video capture board 150 horopta estimation module 152 Disparity image module 154 3D surface reconstruction module 156 noise cancellation and interpolation module 158 3D surface extraction module 160 texture mapping module 280 noise cancellation module 282 smoothing module 284 interpolation module 520 face plane 540 3D face coordinate system formation module 542 3D head Posture estimation module 630 World coordinate system 632 Face coordinate system

Claims (7)

An apparatus for reconstructing 3D human face surface data from a corrected pair of stereo cameras,
Horopter range estimation means for estimating the size and horopter range of the human face based on the calibration parameters of the stereo camera and the corrected image pair;
Disparity estimation means for estimating a disparity value in the horopter range in the other image of the image pair for each pixel of the human face in one image of the image pair;
3D surface reconstruction means for reconstructing 3D surface data of the human face based on the corrected image pair, the disparity value, and the calibration parameters of the stereo camera;
Noise erasing means for erasing noise in the three-dimensional surface data by finding invalid data points having invalid disparity values and interpolating the invalid data points with adjacent data points;
A fitting means for fitting a face plane to the three-dimensional surface data using a predetermined fitting algorithm;
An extraction means for extracting, as a three-dimensional human face, the three-dimensional surface data within a predetermined distance from the face plane. The horopter range estimation means includes
First identifying means for identifying an interocular pattern in the one image of the image pair;
A second specifying means for specifying a corresponding pattern along an epipolar line corresponding to the interocular pattern in the other image;
Verification means for verifying whether a face candidate near the interocular pattern in the one image and the corresponding pattern satisfy a verification condition;
Means for repeatedly operating the first and second specifying means and the verification means until a face candidate satisfies the authentication condition;
The apparatus according to claim 1, further comprising: means for defining a face area by the face candidates satisfying the verification condition. The disparity estimation means includes
Means for defining a correlation window size for the horopter range and the size of the face region;
Means for calculating a disparity value for each pixel in the person's face by searching the other image for a pixel that yields the highest similarity measure according to a pre-selected similarity measure. The apparatus according to claim 2. The noise canceling means is
Means for defining a deviation window size for the size of the face and determining a standard deviation of pixel depth in the deviation window;
For each pixel in the face area in the one image, using the disparity calculated for each of the pixels in the face area in the one image and the image pair, the size near each pixel Means for calculating the local standard deviation and average of the pixel depth within the deviation window of
Each pixel in the face region depends on whether the deviation of the depth value of each pixel from the average deviation window near each pixel is less than the standard deviation value of the deviation window near each pixel. For classifying the
And means for interpolating the depth values of pixels classified as invalid with the depth values of adjacent pixels respectively. The fitting means is
Means for finding an eye line connecting the left eye and the right eye in the three-dimensional surface data;
Means for finding a three-dimensional face symmetry line orthogonal to the line of sight;
Means for selecting data points to be applied to the face plane ;
An apparatus according to any of claims 1 to 4, comprising means for fitting the face plane to the data points to be fitted using the predetermined fitting algorithm. The fitting means is
Means for fitting the face plane to the data points to be fitted using a least squares error fit;
Means for determining whether all errors between the face plane and the three-dimensional surface data are within a predetermined threshold;
Means for eliminating certain high error points from the three-dimensional surface in response to determining that all of the errors are not less than the predetermined threshold;
Means for repeatedly operating the fitting means, the determining means, and the erasing means until the determining means determines that all of the errors are less than the predetermined threshold. 5. The apparatus according to 5. When run on a computer, the computer
A horopter range estimating means for estimating the size and horopter range of a human face on the basis of the calibration parameters and auxiliary Tadashisa image pairs of the scan Tereokamera,
Disparity estimation means for estimating a disparity value in the horopter range in the other image of the image pair for each pixel of the human face in one image of the image pair;
3D surface reconstruction means for reconstructing 3D surface data of the human face based on the corrected image pair, the disparity value, and the calibration parameters of the stereo camera;
Noise erasing means for erasing noise in the three-dimensional surface data by finding invalid data points having invalid disparity values and interpolating the invalid data points with adjacent data points;
A fitting means for fitting the face plane to the three-dimensional surface data using a predetermined fitting algorithm;
A computer program that functions as extraction means for extracting, as a three-dimensional human face, the three-dimensional surface data within a predetermined distance from the face plane.