JP2006146571A - Method of locating and tracking nose bridge and nose tip - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a human nose bridge tracking device suited for real-time tracking in terms of calculation. <P>SOLUTION: A method of locating a nose bridge of a face image includes the step 60 of receiving an image frame; the step 62 of locating eye locations; the step 64 of estimating a location and a size of a face image including the nose and the mouth based on the eye locations and their distance; and the steps 66-72 of determining a nose bridge area in the face image based on intensity distribution in the face image. By carrying out the above method for each consecutive frame, a nose bridge and nose tip can be tracked. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to real-time human nasal bridge and nose tip tracking techniques, along with other facial features in various applications such as visual human recognition, 2D / 3D virtual reality, advanced human-computer interface, and robotics applications. Can be used.

  Traditionally, computers and robots have been considered tools that can perform tasks assigned by human command. Recently, we have come up with symbiotic systems where machines move in this world in a manner similar to humans, especially where they interact as peers in a mutually beneficial relationship with other humans. .

  In such man-machine exchanges, interaction between humans and machines by a plurality of methods is indispensable. In particular, tracking of facial expressions is essential.

  Prior art facial feature tracking algorithms are based on a collection of features based on templates or edges, making initial settings and feature tracking difficult. The reason is that learning of features is required and there are many candidates that can be matched in the tracking process [see Non-Patent Documents 1 and 2]. Furthermore, such an approach requires more computation time. In order to be practical, the facial feature tracking method must operate in real time, be available everywhere, be accurate and robust. In particular, the system must operate under natural conditions for any user, without the need for training, and be able to cope with various facial features, such as when glasses or beards are present.

  The most dominant facial features are said to be the eyes, nose (nasal bridge, nose tip, nostril), and mouth. Although there are many studies on face, eye and mouth tracking [see Non-Patent Documents 3, 4, 5, 6, 7], there is little to see in the automated tracking of the nose tip and nostril. To the best knowledge of the inventors, there is no research on tracking of the nasal bridge.

  Patajan uses the nostril as a reference point for mouth tracking in his work [see Non-Patent Document 2]. Nasal detection is very stable when they are visible. However, depending on the orientation of the face, they are blocked. Therefore, using the nostril as a reference point is generally not a practical approach.

  Kawato describes in Patent Document 1 a method for detecting the tip of the nose no matter what direction the face is facing. Here, an area having the highest light intensity is searched for in the search area estimated from the positions of the two eyes.

  Depending on the lighting setting, this method has been reported to detect the highest light intensity region of the cheek region rather than the nose tip region. In addition, depending on the shape that can be seen in the image frame, a part of the nasal bridge region may have a light intensity higher than that of the tip of the nose, which may disturb the detection accuracy.

  The Gorodniki and Ross nose tip tracking method is based on template search, and a template vector is selected near the extremum of the nose surface [see Non-Patent Document 8]. Thereafter, the pixel pattern recorded during the learning session is tracked by an algorithm in subsequent image frames.

  This method has limitations in the following three points.

  1) A template recording session is required.

  2) Sensitive to lighting conditions. For this reason, the movement of the user's head is limited within a certain range.

3) Tracking subsequent frames depending on the previous tracking results.
JP 2004-157778 A US Pat. No. 4,975,960 Dmitri O. Gorodniki, Gerhard Ross, “Nous, Using Your Nose as a Mouth, Perceptual Vision Technology for Hands-Free Games and Interfaces”, Image and Visual Computing, Vol. 22, 2004. (Dmitry O. Gorodnichy, and Gerhard Roth: Nouse 'use your nose as a mouse' perceptual vision technology for hands-free games and interfaces, Image and Vision Computing, vol. 22, 2004.) Marguerite Becke, James Zips, and Peter Fleming, “Camera Mouse: Visual Tracking of Physical Features to Provide Computer Access to Severely Disabled”, IEEE Neurology and Rehabilitation Engineering, Vol. 10, First issue, March 2002. (Margrit Betke, James Gips, and Peter Fleming: The Camera Mouse: Visual Tracking of Body Features to Provide Computer Access for People with Severe Disabilities, IEEE Transactions on Neural and Rehabilitation Engineering, Vol. 10, No. 1, March 2002.) Shinjiro Kawato and Nobuji Tetsutani, “Real-time Scale Adaptive Face Detection and Tracking with SSR Filter and Support Vector Machine”, ACCV Proceedings Vol. 1, 2004. (Shinjiro Kawato, and Nobuji Tetsutani: Scale Adaptive Face Detection and Tracking in Real Time with SSR Filter and Support Vector Machine, Proc. Of ACCV, vol. 1, 2004.) Chen-Chin Chang, Wen-Kai Tai, Mau-Tseng Yang, E-Tin Han and Chi Jiang Han, “A New Method for Detecting Lips, Eyes and Face in Real Time”, Real-Time Image Processing , Volume 9, 2003. (Cheng-Chin Chiang, Wen-Kai Tai, Mau-Tsuen Yang, Yi-Ting Huang, and Chi-Juang Huang: A Novel Method for Detecting Lips, Eyes and Faces in Real-Time, Real-Time Imaging Vol. 9, 2003.) Yen-Li Tian, Takeo Kanade and Jeffrey F. Cone, “Robust Lip Detection Combining Shape, Color and Motion”, ACCV'2000 Proceedings, Taipei, Taiwan, January 2000. (Ying-li Tian, Takeo Kanade, and Jeffrey F. Cohn: Robust Lip Tracking by Combining Shape, Color and Motion, in Proc. Of ACCV'2000, Taipei, Taiwan, January 2000.) Sabri Grubbs, Keisuke Kinoshita and Sumio Yano, “Mouth Tracking from Video Sequences Using a Trainable Multivariable Gaussian Classifier”, PRMU 2003, Sendai, Japan, December 2003. (Sabri Gurbuz, Keisuke Kinoshita, and Sumio Yano: Mouth Tracking from Video Sequences using Trainable Multivariate Gaussian Classifiers, PRMU 2003, Sendai, Japan, December 2003.) Nuria Oliver, Alex Pentland and François Belard, “Rafter: Real-time face and lip tracker with facial expression recognition”, Pattern Recognition, 33: 1369-1382, 2000. (Nuria Oliver, Alex Pentland, and Francois Berard: Lafter: A Real-time Face and Lips Tracker with Facial Expression Recognition, Pattern Recognition, 33: 1369-1382, 2000) Dmitri O. Gorodniki, “On the importance of the nose in face tracking”, Proceedings of the International Conference on IEEE Automatic Face and Gesture Recognition (FG'02), Washington DC, May 20-21, 2002. (Dmitry O. Gorodnichy: On Importance of Nose for Face Tracking, In Proc. IEEE International Conference on Automatic Face and Gesture Recognition (FG'02), Washington DC, May 20-21, 2002.) Paul Biola and Michael Jones, “Robust Real-time Object Detection”, Second International Workshop on Statistical and Computer-Related Theories on Visual Modeling, Learning, Computing and Sampling, Vancouver, Canada, 13 July 2001 . (Paul Viola, and Micheal Jones: Robust Real-time Object Detection, Second International Workshop on Statistical and Computational Theories of Vision- Modeling, Learning, Computing, and Sampling, Vancouver, Canada, July 13, 2001.)

  An object of the present invention is to provide a human nasal bridge tracking device that is computationally suitable for real-time tracking.

  Another object of the present invention is to provide a nasal bridge tracking device that is robust to shade and illumination changes.

  It is a further object of the present invention to provide a nasal bridge tracking device that is invariant to changes in face orientation and scale as seen from the front.

  Another object of the present invention is to provide a nasal bridge tracking device that is invariant to skin color and facial dimensions.

  Accordingly, another object of the present invention is to provide a nasal bridge tracking device based on the photometric attributes of the nasal bridge without using any assumptions on the person and without requiring any kind of training data about the person. Is to provide.

  Other objects, features and advantages of the present invention will become apparent to those skilled in the art of pattern analysis and recognition from the following summary, drawings and detailed description of the present invention and preferred embodiments.

  The method of the present invention improves the detection rate and detection accuracy of the nose tip position, and also detects the nasal bridge line. This starts from the end of the nasal bridge closer to the eye and searches downward until the tip of the nose is found. Thus, the nose bridge and the nose tip position can be output in real time.

  A method for tracking a person's nasal bridge in real time in the presence of changes in geometry and illuminance includes obtaining an image frame from a color or monochrome video device as an input image, and extracting the currently input image. Processing to determine the position of the eye, estimating the position and size of the face in the image based on the distance between the eyes, and the general face area of interest, including the nose and possibly the mouth area Cutting out (region of interest: ROI) and outputting the nose bridge line and the tip of the nose in real time based on the photometric attributes of the nose bridge.

  More specifically, according to one aspect of the invention, a computer-implemented method for locating a nasal bridge in a facial image includes receiving an image frame from a video image generating device, and an image frame Identifying the position of the eye in the middle image, estimating the position and size of a face image including the nose and mouth of a face based on the position of the eye and its spacing, and light of the face image Determining a nasal bridge area in the face image based on the intensity distribution.

  Preferably, the determining step includes calculating a horizontal light intensity profile for each row segment of the face image and finding a nose bridge candidate in the horizontal light intensity profile for each row segment of the face image, The light intensity profile satisfies a predetermined criterion at the position of the nose bridge candidate.

  More preferably, the step of finding includes applying a predetermined maximum filter to the horizontal light intensity profile for each row segment of the face image, the maximum filter including three segments, and further comprising a row of the face image For each of the segments, find a nostril candidate that satisfies S1 ≦ S2 ≧ S3, where S1, S2, and S3 are the light intensity values of the leftmost segment, the central segment, and the rightmost segment, respectively, in the maximum filter. The maximum filter further sets A, B, and C as positive constants so that the weighted sum, AS1 + BS2 + CS3, is maximum in the row segment.

  Constant B may be greater than constants A and C.

  The constant A may be equal to the constant C.

  According to another aspect of the present invention, a computer-implemented method for tracking the nasal bridge of a facial image in an image sequence is performed using any of the methods described above for each facial image. Locating the nostril of the nasal bridge.

  According to yet another aspect of the present invention, a computer-implemented method for finding the tip of the nose from a facial image includes performing all steps of any of the methods described above and determining. In the determined nasal bridge area, the step of identifying the maximum intensity point where the light intensity value of the face image is highest in the nasal bridge area, and the column segment in the face image extending from the maximum intensity point to the lower end of the face image are defined. A step of creating a light intensity profile of the row segment along the row direction of the row segment, and investigating the first and second derivatives of the light intensity profile, A step of identifying a level, a step of forming a nasal bridge line by obtaining a regression line of a nasal bridge candidate, As, and a step of finding the nose tip.

  Preferably, the step of identifying the nose tip level corresponds to the tip of the nasal bridge area in the light intensity profile of the row segment by identifying the point where the first derivative of the light intensity profile of the row segment has a maximum value. Identifying a level to perform and identifying a nostril level in the light intensity profile of the row segment by identifying a point where the second derivative of the light intensity profile of the row segment has a predetermined value; Determining the level of the nose tip as a predetermined point between the tip level of the nasal bridge area and the nostril level.

  Determining the level of the nasal tip includes calculating the average of the tip of the nasal bridge area and the nostril level in the row segment.

  The predetermined value may be zero.

  According to another aspect of the present invention, a computer-implemented method for tracking the nose tip of a facial image in an image sequence is performed using any of the methods described above for each facial image. Identifying the position of the nose tip of the image.

  When the computer program of the present invention is executed on a computer, it causes the computer to perform all the steps of any of the methods described above.

[Overview of the embodiment]
The embodiment of the invention described below overcomes the previously recognized problems associated with nasal tracking by combining the photometric attributes of a human nose curve with eye tracking technology. Initially, the position of the eye is found to estimate the ROI of the face, after which the method utilizes the light intensity profile of the nasal curve along both the nasal bridge and the nose tip. In the proposed approach, the nasal bridge tracking device is robust to changes in lighting conditions, skin tone, and facial geometry as well as facial movements as seen from the front.

  FIG. 1 is an overview block diagram of a tracking device 30 according to this embodiment for tracking the nasal bridge and tip of the nose. Referring to FIG. 1, a tracking device 30 stores a color (or monochrome) video camera 40 for capturing an image frame sequence of a human face at a predetermined speed, and each frame captured by the video camera. A frame memory 42 for detecting the position of the eye in each frame stored in the frame memory 42, a nose bridge on the face of the person photographed by the camera 40, and a nose bridge line And a nose bridge detection module 46 for outputting nose tip position data.

  For each frame, the tracking device 30 of this embodiment obtains an image frame from the video camera 40. The eye detection module 44 detects the eye position of each frame using the light intensity plane. That is, the nose bridge detection module 46 uses the relative light intensity information (light intensity distribution) of the curve of the human nose in combination with the eye tracking technique briefly described in the section entitled “Eye Detection and Tracking”. To do. In this embodiment, the captured images are red, green, and blue (red-green-blue: RGB) images. Thus, the nose bridge detection module 46 saves computation time by using one of the color planes as a measure of light intensity information. In this embodiment, a green plane is used as representative of the light intensity information.

  Thus, the eye which is the dominant feature of the face is first tracked as a reference point by the eye detection module 44.

  Thereafter, the nasal bridge and the tip of the nose are detected by the nasal bridge detection module 46 in real time in each image frame without depending on the past tracking results. Algorithms used in the eye detection module 44 and the nasal bridge detection module 46 will be described later.

-Eye detection and tracking-
The tracking device 30 of this embodiment is realized by computer hardware and a specific computer program executed on the computer. FIG. 2 shows the overall control structure of the computer program. Prior to describing the overall structure of FIG. 2, the details of each step of FIG. 2 will be described with reference to FIGS.

  With reference to FIG. 3, facial eye positions 122 and 124 in the captured image 120 are first detected. The eye detection and tracking algorithm used in this step is described in detail in Non-Patent Document 3.

  Next, a face region of interest (ROI) 126 is extracted from the image 120. In simple terms, the pattern between both eyes is detected and tracked by updated template matching. In order to cope with changes in the size of the face, images of various scales are considered for detection, and an appropriate scale is selected according to the distance between both eyes.

  The algorithm calculates an intermediate representation of the input image called “Integral image” described in Non-Patent Document 9. Thereafter, the light-dark relationship of the eye region in the image is filtered using a 6-segment orthogonal (SSR) filter. The resulting face candidates are further verified by a support vector machine (SVM).

-Detection of nasal bridge and tip of nose-
The face ROI 140 is selected as described in the section entitled “Eye Detection and Tracking”. With reference to FIG. 3, the ROI selection process of the face using the eye positions 122 and 124 in the captured image 120 is approximately 1.2 × distance between both eyes and 1.8 × both eyes. Selecting a rotation-corrected rectangular region 126 having a height dimension between the distances. The resulting ROI candidate 126 is cut out and rotated to obtain an image of the face ROI 140. The face ROI 140 starts at the eye level defined by the eye positions 142 and 144 and typically extends to the upper neck.

  The human nose has a convex shape, and the nasal bridge from the eye level to the tip of the nose extends in a line. In general, the light intensity measurement at a point decreases as the distance between that point and the light source increases. This is because light spreads away from the position of the light source. Therefore, information on the curvature is expressed by a light intensity distribution. FIG. 4 shows the light intensity profile 170 of the row segment 160 extracted from the face ROI 140 on the right side.

  In this embodiment, this physical phenomenon is used in connection with noise reduction and pattern search.

  The nose has a convex shape. Therefore, more light intensity is gathered at the nose bridge than at the side. Instead of using a single row of the face ROI image for the light intensity profile, video noise is obtained by using multiple rows to sum the light intensity values of the vertical lines of the pixels indicated by the vertical rectangle 162. Can significantly reduce the impact of Thus, at the same time that the effects of noise are cancelled, the light intensity values are accumulated at a faster rate in the nasal bridge than in the side of the nose. By repeating this process for segments having overlapping portions below the eye line, a 3D pattern 170 of the light intensity profile is generated.

  That is, the point on the convex curved surface has higher light intensity than the side portion. In this embodiment, a fast filtering approach is used to trace the nasal bridge using the light intensity distribution of the row segment 160.

Similar to the interocular detection filter described in Non-Patent Document 3, in this embodiment, a 3-segment convolution filter called a maximum filter is used, and the position of the nostril candidate is determined using the integrated value of the light intensity. Tracing. Here, the central segment is equal to or larger than the side segment, and the sum of the light intensities multiplied by the constants of all three segments is the maximum value at the position of the nose bridge. 4 and 5 show a maximum filter 180 with three segments 182, 184 and 186 that trace the maximum light intensity pattern for each row starting from the eye line. The criteria are as follows.

ここでＳ_ｉは図５に示す最大フィルタのセグメント１８２、１８４及び１８６の光強度の積分値を示し、Ａ、Ｂ及びＣは全て予め定められた定数である。好ましくは、定数Ｂは定数Ａ及びＣより大きい。定数Ａは定数Ｃと等しくてもよい。例えば、Ｂ＝３でありＡ＝Ｃ＝１である。

Here, S _i represents the integrated value of the light intensity of the segments 182, 184 and 186 of the maximum filter shown in FIG. 5, and A, B and C are all predetermined constants. Preferably, constant B is greater than constants A and C. The constant A may be equal to the constant C. For example, B = 3 and A = C = 1.

  The filter is convolved with all row segments of the face ROI image. In each row, the position of the pixel that satisfies the criterion of Equation 1 is considered to be a nose bridge candidate to be further processed.

Direct convolution of the maximum filter with the row segments of the image is a computationally expensive process. Thus, with reference to FIG. 6, using the cumulative total of 200 of the light intensity values in a row segment, the S _i can be calculated using the two reference indices of S _i. The cumulative total for each row segment is calculated as follows:
CSUM (0) = rowArr (0),
CSUM (j) = CSUM (j−1) + rowArr (j); j = 1, 2,... N (2)
Here, rowArr and CSUM are the light intensity value of the row array and its accumulator, respectively. Therefore, S _i between indices k ₁ and k ₂ can be calculated as follows:
S _i (k ₁ ; k ₂ ) = CSUM (k ₂ ) −CSUM (k ₁ ) (3)
Each row segment is processed until either end of the face ROI is reached or Equation 1 is not satisfied to find the nasal bridge position using Equation 1. That is, as shown in FIG. 7, when nasal bridge candidate points 220 are found, some of them 222 are not on the nasal bridge and may be lower than the tip of the nose. For this reason, as shown by the box 240 in FIG. 8, a correction process is required so that the nose bridge candidate is positioned on the nose bridge.

  T (r) represents the maximum accumulated light intensity total at the j-th position of the r-th row segment of the face ROI image output from the maximum filter 180, defined by the following equation.

ここでＴ（ｒ）は鼻梁上の３セグメントフィルタの区域下の光強度の累算に相当する。従って、拡散光の条件下で、或る点と光源との距離が増加すれば、又はその点が陰になれば、その点のＴ（ｒ）値は小さくなる。このことから、鼻先端は鼻孔線上の点よりも高いＴ（ｒ）を有することが分る。なぜなら、鼻孔線上の点は、図８の右側のＴ（ｒ）プロファイル線２４２で示されるように、鼻曲線上にはないからである。このため、鼻先端と口ひげの線との間ではＴ（ｒ）の勾配が高いことが期待され、さらに、鼻孔線上方の全ての候補点は鼻孔線上のものより高いＴ（ｒ）を有するものと期待される。

Here, T (r) corresponds to the accumulation of light intensity under the area of the three-segment filter on the nose bridge. Therefore, if the distance between a certain point and the light source increases under the condition of diffused light, or if that point becomes shaded, the T (r) value at that point decreases. From this it can be seen that the nose tip has a higher T (r) than the point on the nostril line. This is because the point on the nostril line is not on the nose curve as indicated by the T (r) profile line 242 on the right side of FIG. Therefore, it is expected that the gradient of T (r) is high between the nose tip and the mustache line, and all candidate points above the nostril line have a higher T (r) than that on the nostril line. It is expected.

-Algorithm for obtaining nose bridge line and tip-
The algorithm for obtaining the nasal bridge line and nose tip is described below.

  1. The gradient ∇T (r) of the T (r) line 242 is determined. In this embodiment, ∇T (r) is determined as shown in FIG. Referring to FIG. 9, ∇T (r) is equal to the difference between T (r) (eg, at point 260) and T (r + 1) (eg, at point 262) divided by Δ. Here, Δ is a predetermined natural number, for example, 3. According to this definition, ∇T (r) becomes positive if the value of T (r) decreases. The absolute value indicates the steepness of the curve. Of course, other definitions may be used.

  2. Sort the indices of ∇T (r) in descending order of gradient values.

  3. Assuming that the row index starts at the eye level, find the kth row starting from the highest gradient position of ∇T (r), where T (r) ≧ T (k) ∀r <k. This means that the point at which the value of T (r) sharply decreases is considered the earliest.

  4). Referring to FIG. 10, the index i of the row where T (i) is the maximum for i <k is found, and the column segment starting from the i-th row is selected. FIG. 10 shows a column segment. Referring to FIG. 10, column segment 270 is a vertical segment of the face ROI, starting from the i th row and typically extending to the lower end of the face ROI.

  5. Referring to FIG. 11A, the horizontal light intensity profile (hproj) of the column segment 270 in the vertical direction is formed.

  6). Referring to FIG. 11B, the influence of video noise on hproj 280 is removed to obtain a smoothed light intensity profile (shproj) 282. For example, a fast Fourier transform (FFT) is performed on hproj, all FFT coefficients above 0.15π digital frequency are set to zero, and an inverse fast Fourier transform (IFFT) of the resulting array of coefficients is taken. Look for the first three extreme values (two maxima 284 and 288 and one minima 286) on the smoothed profile.

  7. Referring to FIG. 11 (c), the absolute value (| ｊshproj |) of the first derivative 294 at the tip of the nose has a maximum value 296 between the first maximum 284 and the first minimum 286. .

8). Still referring to FIG. 11 (c), the first derivative is either zero or a nostril constant. That is, the first point 298 where the second derivative is zero (∇ ² shproj (p) = 0) after the nose tip is the nostril line.

  9. Furthermore, as shown by a line 312 in FIG. 12 from the nose bridge to the tip of the nose, a regression line of all points extracted using Equation 1 is obtained to represent the nose bridge. In FIG. 12, the regression line from the nostril points 310A to 310N is obtained. Line 312 is assumed to represent the nasal bridge.

  10. Referring to FIG. 12, the nose tip is some point between the nostril tip on the nostril line 312 and the nostril line found in the previous step. Therefore, the nose tip is assumed to be on a certain line. Line 314 in FIG. 12 represents this line. The tip of the nose is an intersection 316 between the nose bridge line 312 and the line 314.

-Realization by computer-
The nasal bridge and nasal tip tracking device 30 of this embodiment do not require fine adjustment of parameters depending on skin color differences and illumination conditions. Regardless of which user appears in front of the camera at any time, the tracking device 30 automatically starts tracking the person's eyes, nasal bridge and nose tip without learning.

  Returning to FIG. 2, the overall control structure of the program for realizing the apparatus 30 of this embodiment will be described. The program begins at step 60. For each frame, system 30 captures an image frame at step 60. In step 62, eyes are detected in the image taken in step 60. At step 64, the face ROI is extracted.

  Next, steps 66 to 72 are repeated for each row segment of the extracted face ROI. Specifically, for each row segment, the cumulative sum of all pixel columns in the row segment is calculated in step 68, and the maximum filter (see FIG. 5) is applied to the cumulative sum calculated for the target row segment in step 70. Thus, a horizontal T (r) profile of the row segment (see FIG. 4) is obtained. An example of the T (r) profile thus obtained is shown on the right side of FIG.

  For each row segment, a T (r) profile is obtained in steps 68 and 70, and then a mustache line is estimated in step 74. FIG. 8 and FIG. 9 show how to estimate the mustache line. At step 76, the largest T (r) row above the mustache line is selected. Step 78 extracts column segments that extend to the bottom of the face ROI under the row selected in step 76. An example of a column segment is shown as column segment 270 in FIG.

  At step 80, a horizontal projection of the column segments extracted at step 78 is formed. An example of this projection is shown in FIG. In step 82, the projection is smoothed by applying an FFT to the projection, setting the FFT coefficients above a predetermined digital frequency to zero, and taking the IFFT of the resulting array of FFT coefficients. . An example of the smoothed projection is shown in FIG.

  At step 84, the first three extreme values of the profile (the two maxima and the minima between them) are found as points 284, 286 and 288 as shown in FIG. 11 (b). At step 86, the first derivative of the profile between points 284 and 286 is taken, and the maximum point of the first derivative indicates the tip of the nasal bridge, which is point 296 in FIG. 11 (c).

  At step 88, the second derivative of the profile is taken and the point where the second derivative is zero after point 296 indicates the nostril line. The tip of the nose is somewhere between the tip of the nasal bridge and the nostril line. In this embodiment, the nose tip level is calculated in step 90 as the average (center point) of the tip of the nasal bridge and the nostril position.

  In step 92, regression lines (nasal bridge lines) of all points on the nasal bridge are obtained. In step 94, the intersection of the nose tip line and the nose line is searched. This is the tip of the nose.

  FIG. 13 shows an overview of a computer system 330 that executes the above-described program to realize the apparatus 30 of this embodiment. The above-described embodiment is realized by computer hardware and a computer program executed on the computer hardware. FIG. 13 is an external view of the computer system 330 of this embodiment, and FIG. 14 shows the system 330 in a block diagram.

  Referring to FIG. 13, a computer system 330 includes a computer 340 including an FD (Flexible Disk) drive 352 and a CD-ROM (Compact Disc Read Only Memory) drive 350, a keyboard 346, a mouse 348, a monitor 342, And a video camera 40.

  14, in addition to the FD drive 352 and the CD-ROM drive 350, the computer 340 includes a CPU (Central Processing Unit) 356 and a bus 366 connected to the CPU 356, the CD-ROM drive 350, and the FD drive 352. A read-only memory (ROM) 358 for storing a program such as a boot-up program, and a random access memory connected to the CPU 356 for temporarily storing instructions of the application program and providing a temporary storage space. (RAM) 360 and a hard disk 354 for storing application programs, system programs, and data. Although not shown here, the computer 340 may further include a network adapter board that provides a connection to a local area network (LAN).

  A program for causing the computer system 330 to execute the function of the tracking device 30 of this embodiment is stored in the CD-ROM 362 or FD 364, inserted into the CD-ROM drive 350 or FD drive 352, and further transferred to the hard disk 354. Also good. Alternatively, the program may be transmitted to the computer 340 via a network (not shown) and stored in the hard disk 354. The program is loaded into the RAM 360 when executed. The program may be loaded directly from CD-ROM 362, FD 364, or network.

  The program includes several instructions for causing the computer 340 to execute the function of the tracking device 30 of this embodiment. Since some of the basic functions required are provided by an operating system (OS) or a third party program running on the computer 340, or a module installed on the computer 340, the program is of this embodiment All the basic functions for realizing the tracking device 30 are not necessarily included. The program only needs to include the part of the instruction that calls the appropriate function in a controlled manner and obtains the desired result. How the computer system 330 operates is well known and therefore will not be repeated here.

-Tracking results-
Figures 15 and 16 show some examples of automatically tracked nasal bridges and tips. As shown in FIG. 15, the nose tips 380, 382 and 384 are accurately tracked under different orientations and different illumination settings. Further, as shown in FIG. 16, the nose tip 390 can be tracked even when wearing sunglasses.

  The proposed algorithm is implemented in C ++ and runs under a normal OS. Experimental results verified that the device 30 implemented with computer hardware and software was executed in real time, fully automatically, at 30 frames per second (fps). The device 30 successfully tracked an individual regardless of (1) different skin colors, (2) face orientation and scale changes as seen from the front, and (3) lighting changes.

  In this embodiment, a convex curved surface tracking filter is used for tracking the nasal bridge. The above description also shows that the first derivative of the light intensity profile at the tip of the nasal bridge has a maximum value. Similarly, the second derivative of the light intensity profile is zero at the nostril line. The nose tip is between these two levels.

  The apparatus and method of the present invention has been described in connection with nasal bridge tracking. However, various modifications and changes are possible to track other facial feature points.

  In the above-described embodiment, the image is a color, but the present invention is not limited to such an embodiment, and the image may be only a light intensity image. In that case, the light intensity information itself may be used directly as a representation of the light intensity information.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the embodiment described above. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a block diagram of tracking device 30 of one embodiment of this invention. It is a flowchart of the program run on the computer which implement | achieves the tracking apparatus 30 of this embodiment. It is a figure which shows extraction of face ROI. It is a figure which shows how the line segment 160 is extracted from the face ROI. It is a figure which shows the 3 segment convolution filter 180 used by this embodiment. It is a figure which shows the cumulative sum 200 of the light intensity value of a row segment. FIG. 7 shows successfully extracted nasal bridge candidates and some false nasal bridge candidates that were accurately found by further processing of the nasal bridge and nose tip. FIG. 6 is a diagram showing successfully extracted nasal bridge candidates (some of which are below the nasal tip and therefore not accurate) and their light intensity profiles. It is a figure which shows the definition of ∇T (r). It is a figure which shows a column segment. It is a figure which shows the profile of a column segment. It is a figure which shows how a nose tip is determined. It is a figure which shows the external appearance of the computer system 330 which runs the above-mentioned program and implement | achieves the apparatus 30 of this Embodiment. It is a figure which shows the structure of the computer 340 shown in FIG. It is a figure which shows the experimental result of this embodiment. It is a figure which shows the experimental result of this embodiment.

Explanation of symbols

30 Tracking device 40 Video camera 42 Frame memory 44 Eye detection module 46 Nasal bridge detection module 122, 124, 142, 144 Eye position 126, 140 Face region of interest (ROI)
160 row segment 170 light intensity profile 180 maximum filter 270 column segment 280 light intensity profile (hproj)
282 Smoothed light intensity profile (shproj)
330 Computer System 340 Computer

Claims (9)

A computer-implemented method for locating a nose bridge in a facial image,
Receiving an image frame from a video image generating device;
Identifying the eye position of the image in the image frame;
Estimating the position and size of a facial image including the nose and mouth of the face based on the position of the eye and its spacing;
Determining a nostril area in the face image based on a light intensity distribution of the face image. Said determining step comprises:
Calculating a horizontal light intensity profile for each row segment of the facial image;
Finding a nasal bridge candidate in the horizontal light intensity profile for each of the row segments of the facial image, wherein the horizontal light intensity profile satisfies a predetermined criterion at the position of the nasal bridge candidate. The method according to 1. The step of finding comprises
Applying a predetermined maximum filter to the horizontal light intensity profile for each of the row segments of the facial image, the maximum filter including three segments;
For each of the row segments of the face image, the method includes finding a nostril candidate that satisfies S1 ≦ S2 ≧ S3, where S1, S2, and S3 are respectively the leftmost segment, the middle segment, and the rightmost segment in the maximum filter. 3. The light intensity value of each of the segments, and wherein the maximum filter further sets A, B, and C to be positive constants so that the weighted sum AS1 + BS2 + CS3 is maximized in the row segment. Method. A computer-implemented method for tracking a nasal bridge of a facial image in an image sequence, wherein for each of the facial images, a face is utilized using the method according to any of claims 1-3. Locating the nasal bridge of the image. A computer-implemented method for finding the tip of the nose from a facial image,
Performing all the steps according to any one of claims 1 to 3,
In the nasal bridge area determined in the determining step, identifying a maximum intensity point at which the light intensity value of the face image is highest in the nasal bridge area;
Defining a column segment in the face image extending from the maximum intensity point to a lower end of the face image;
Creating a light intensity profile of the column segment along a column direction of the column segment;
Determining the level of the tip of the nose by examining the first and second derivatives of the light intensity profile;
Forming a nasal bridge line by determining a regression line of the nasal bridge candidate; and
A method for finding the tip of the nose from a face image, comprising the step of finding the tip of the nose as an intersection of the line defined by the level of the tip of the nose and the nose bridge line. Identifying the level of the tip of the nose,
Identifying a level corresponding to the tip of the nasal bridge area in the light intensity profile of the row segment by identifying a point where the first derivative of the light intensity profile of the row segment has a maximum value;
Identifying a nostril level in the light intensity profile of the row segment by identifying a point where the second derivative of the light intensity profile of the row segment has a predetermined value;
6. The method of claim 5, comprising determining a nasal tip level as a predetermined point between the nasal bridge segment tip level and the nostril level. The method of claim 6, wherein determining the level of the nose tip comprises calculating an average of the tip of the nasal bridge area and the nostril level in the row segment. A computer-implemented method for tracking the nose tip of a facial image in an image sequence, wherein each of the facial images utilizes the method according to any of claims 5-7. A method for tracking a nasal tip of a facial image in an image sequence comprising the step of locating the nasal tip of the facial image. A computer program that, when executed on a computer, causes the computer to perform all the steps according to any one of claims 1 to 8.

Images