JP2003141552A - Object direction calculation device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pointer display device for rapidly and accurately calculating face direction from images inputted one after another by accurately determining and registering the three-dimensional configurations of the characteristic points of the face, and utilizing the general inverse matrix of the registered configuration matrix. SOLUTION: Images of users with various face directions are inputted by an image input part 1. A characteristic point detecting part 2 detects P characteristic points of each image and obtains a measurement matrix W therefrom. Next, a configuration matrix registering part 3 determines a configuration matrix S from the measurement matrix W and registers it. When a particular user uses a pointer on a personal computer, the direction of the user's face needs to be detected. An image of the user's face is taken and inputted by the image input part 1 while the user is explaining. P characteristic points are detected from the image and a measurement matrix Wnew is obtained therefrom. The general inverse matrix is obtained from the registered configuration matrix S and a movement matrix Mnew showing the face direction is obtained from the general inverse matrix and the measurement matrix Wnew .

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention inputs a moving image captured by a CCD camera or the like, calculates the three-dimensional orientation of the face in real time from the image, and determines the position of the face on the display. Related to the technical field to display.

[0002]

2. Description of the Related Art When operating a personal computer or the like, a mouse is widely used to move a cursor on a display. For the operator, moving the hand placed on the keyboard to move the cursor to the mouse is an operation that impedes smooth work flow. On a notebook PC, you can use the AccuPoint, trackpad, etc. to move the cursor without having to move your hands significantly from the keyboard, but it takes time to get used to it, and it is inconvenient if you move a wide range. Therefore, it is not uncommon to attach a mouse after all.

[0003] Therefore, if the cursor can be moved by moving the face, not only the work placed on the keyboard can be moved efficiently but also the work can be performed over a wide range. You can do it without difficulty. Further, since the direction of the face represents the direction of human attention, it is considered that moving the cursor in the direction of the face is a motion suitable for human intuition.

However, a pointer display technique utilizing such face orientation has not been established yet. In order to realize this, it is necessary to accurately obtain the face orientation in real time.

The conventional face orientation calculation methods are roughly classified into two types.

The first conventional method is a method in which a face image pattern for each orientation is directly or dimensionally compressed as a template, and the orientation corresponding to the most matched template is determined as the orientation of the face by pattern matching of the entire face. .

The second conventional method is a method of detecting feature points of the face such as the eyes, nose and mouth, and geometrically calculating the orientation using the image coordinates of these.

Hereinafter, the first conventional method will be referred to as a pattern-based method, and the second conventional method will be referred to as a feature point-based method.

Since the pattern space method handles face orientation calculation as a pattern classification problem, the face orientation result is output as discrete values, and the accuracy is low as it is. In order to obtain a continuous value, processing such as interpolation is required separately. Such a method is not suitable for application to a pointer or the like, which requires accurate and high-speed face orientation determination.

On the other hand, the feature point-based method is a precise method in which the face orientation can be accurately calculated as continuous values as long as the coordinates of the feature points are accurately obtained.

Among them, a typical excellent method is Toma.
This is a factorization method proposed by Si and Kanade in 1991 (Tomasi. C. and T. Kanad.
e: Technical Report CMU-CS
-91-172, CMU (1991); Interna
regionalJournal of Computer
Vision, 9: 2, 137-154 (199
2)).

With this method, it is not necessary to obtain camera parameters, and it is possible to obtain the three-dimensional shape and the orientation of each frame from the two-dimensional images of a plurality of frames. However, it is useful for batch processing because it calculates multiple frames collectively, but it is not suitable for online processing in which images to be processed are input one after another.

In order to handle this in real-time processing, Fujiki and Kurata proposed a recursive factorization method (Fujiki,
Kurata: Transactions of the Institute of Electronics, Information and Communication Engineers, J84-D-II: 8,
1663-1673 (2001)).

With this method, the processing time can be reduced to 1/10 of that of the factorization method of batch processing, but further speeding up is required if the processing time for feature point detection is also added. Further, since the orientation and the three-dimensional shape obtained for each new frame are used for the orientation calculation of the next frame, an error may be accumulated.

When performing face direction calculation based on feature points, it is necessary to stably detect face feature points.

Conventional methods for detecting feature points are roughly classified into two methods.

The first conventional method is a method in which after corners are detected, correspondence between the corners detected in each frame is obtained between the frames.

In this case, which feature point is obtained depends on changes in environment such as lighting conditions and individuals. In addition, the obtained feature points do not always match the eyes, nose, mouth, etc.

The second conventional method is a method in which the characteristic point to be detected is determined in advance and the characteristic is used for detection. For example, eyes, nose, mouth, etc. are often used as facial feature points.

Since Fukui and Yamaguchi have circular eyes and nostrils, they have shown that the pupils and nostrils can be stably detected using a circular resolution filter (Fukui and Yamaguchi:
IEICE Transactions, J80-D-II: 8,217
0-2177 (1977)).

The end of the mouth, that is, the joint between the upper lip and the lower lip is called the mouth end, and a separability filter can be used at the mouth end. However, since the shape of the mouth end itself is unstable due to individual differences, etc., a separability filter that performs detection using the shape is not well suited.

After the face orientation is obtained, the position where the pointer is displayed has to be calculated from the obtained face orientation result. However, if the user wants to move the pointer with a large facial movement, or conversely, the pointer may move even with a small movement. The user-friendliness criteria differ depending on the user and the display area of the pointer. In order to realize a pointer device that is easy for the user to use, it is conceivable that the range in which the face is moved matches the pointer display area. However, such technology does not exist yet.

[0023]

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to realize a pointer display device using face orientation and to establish elemental technology necessary for that purpose.

The most important task of the elemental technologies is to accurately calculate the orientation of the face in real time from the input moving image.

When the direction of the face is interlocked with the movement of the pointer or the like, it is necessary to obtain the direction of the face as a continuous value.

In the present invention, a feature point-based method that obtains continuous values is adopted instead of the pattern matching method that obtains face directions by discrete values as described in the prior art.

In this case, stable detection of facial feature points is also an important issue. Further, when the display position of the pointer is calculated from the calculation result of the face orientation, it is also an object of the present invention to make the user's intuition easy to use by matching the range of the user's face orientation with the range of movement of the pointer. one of.

[0028]

According to a first aspect of the present invention, there is provided an object orientation calculation device for calculating an orientation of an object, wherein a shape matrix representing a spatial position of a feature point of the object is created in advance,
General inverse matrix calculating means for obtaining a general inverse matrix from this shape matrix, image input means for inputting an image of the object, and characteristic points of the object are detected from the image and measured from these characteristic points. Object direction for calculating a motion matrix expressing the information of the direction of the object by using a feature point detecting means for creating a matrix, a measurement matrix created by the feature point detecting means, and a general inverse matrix for the object created in advance An object orientation calculation apparatus comprising: a calculation unit.

According to a second aspect of the present invention, the general inverse matrix calculating means inputs a plurality of images in which various directions of the object are photographed by the image inputting means, and the feature point detecting means extracts from each of the images. The feature points of the object are respectively detected, and one measurement matrix peculiar to the object is created from these feature points, and the shape matrix peculiar to the object is obtained from this measurement matrix by using the factorization method. The object orientation calculation apparatus according to claim 1, wherein a general inverse matrix is obtained.

According to a third aspect of the present invention, a point on the display to which the human face faces is calculated from the motion matrix representing the human face direction information calculated by the object direction calculating means, and a pointer is placed on the point. 2. The object orientation calculation apparatus according to claim 1, further comprising pointer display means for displaying.

According to a fourth aspect of the present invention, the pointer display means obtains a face normal vector indicating the face orientation from the motion matrix calculated by the object orientation calculation means, and adjusts the coefficient of the normal vector. 2. The object orientation calculation apparatus according to claim 1, wherein the moving range of the pointer is adjusted.

According to a fifth aspect of the present invention, there is provided a feature point detecting device for detecting a feature point of a face, which is based on continuous edges of the mouth and brightness around the mouth whose brightness is brighter than the edges of the mouth. A feature point detection device characterized by detecting a feature point representing a mouth end.

A sixth aspect of the present invention is an object orientation calculation method for calculating the orientation of an object, wherein a shape matrix representing the spatial position of the feature points of the object is created in advance, and a general inverse matrix is created from this shape matrix. A general inverse matrix calculation step for obtaining the object, an image input step for inputting an image in which the object is photographed, feature points of the object are detected from the image, and feature point detection for creating a measurement matrix from these feature points Step, the measurement matrix created in the feature point detection step,
And an object orientation calculation step of calculating a motion matrix representing the orientation information of the object using a general inverse matrix created in advance for the object, the object orientation calculation method.

According to a seventh aspect of the present invention, there is provided a program for realizing an object orientation calculation method for calculating an orientation of an object by a computer, wherein a shape matrix representing a spatial position of a feature point of the object is created in advance, and A general inverse matrix calculation function for obtaining a general inverse matrix from a shape matrix, an image input function for inputting an image in which the object is photographed, a feature point of the object is detected from the image, and a measurement matrix is obtained from these feature points. An object orientation calculation for calculating a motion matrix representing the orientation information of an object using a feature point detection function for creating a measurement matrix created by the feature point detection function, and a general inverse matrix for the object created in advance It is a program of an object orientation calculation method characterized by realizing functions and.

[0035]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a pointer device using the face orientation calculation result of the present invention will be described below with reference to the drawings.

1. Configuration of Pointer Device 20 FIG. 1 shows a pointer device 20 using a face orientation calculation result.
3 is a block diagram showing the configuration of FIG.

The pointer device 20 includes an image input unit 1 for inputting a moving image or a still image from a CCD camera, a feature point detecting unit 2 for detecting feature points in the input image, a shape matrix registering unit 3, and a general matrix. An inverse matrix storage unit 4, and a face orientation calculation unit 5 that calculates a face orientation using the feature points detected by the feature point detection unit 2 and the general inverse matrix of the shape matrix stored in the general inverse matrix storage unit 4. And a pointer display unit 6 for calculating a point on the display where the face is facing from the face direction calculation result of the face direction calculation unit 5 and displaying a pointer on the calculated point.

The function of each of these components is executed by a program stored in the computer.

The image input unit 1, the feature point detection unit 2, and the shape matrix registration unit 3 execute the shape matrix registration method.

Further, the general inverse matrix storage unit 4 and the image input unit 1
The feature point detection unit 2 and the face orientation calculation unit 5 execute the face orientation calculation method.

The feature point detection unit 2 executes the method of detecting the mouth edge and other feature point detection methods such as eyes, nose, and moles.

The pointer display unit 6 executes an automatic adjustment method for pointer display and a user adjustment method for pointer display.

With such a system, the pointer display device 20 utilizing the face orientation is realized by implementing each elemental technique and integrating the results.

A brief description will be given of how the face orientation calculation result is obtained by the pointer device 20.

First, as a preparation, the shape matrix of the user is registered in advance. For this, the image input unit 1 inputs images obtained by shooting various face orientations for each user.

Next, the characteristic point detecting section 2 detects P characteristic points of the image and obtains the measurement matrix W from them.

Next, the shape matrix registration unit 3 obtains and registers a user-specific shape matrix S from this measurement matrix W.

When a specific user uses a pointer on a personal computer, it is necessary to detect the face orientation of the user. At that time, the image input unit 1 captures the face of the user in use and inputs the image. P feature points are detected from this image, and a measurement matrix W _new is obtained from the detected P feature points. Then, a general inverse matrix is obtained from the shape matrix S registered in advance above, and the general inverse matrix and the measurement matrix W are obtained.
A motion matrix M _new representing the face orientation is _calculated from _new .

2. Description of factorization method First, a factorization method for obtaining a shape matrix from a measurement matrix will be described.

Here, an orthographic projection model is assumed for simplicity of explanation, but it can be extended to a weak central projection model, a pseudo central projection model, or the like.

Let F be the number of frames and P be the number of characteristic points. f
The coordinates of the feature point in the frame image are {(u_fp，
v_fp) | P = 1, ..., P}, and the barycentric coordinate is (uC_f，
vC _f). The barycentric coordinates were subtracted from the coordinates of the feature points
Things

[Equation 1]

[Equation 2] And the 2F × P matrix is defined by W = [w _ij ]. This is called the measurement matrix. The measurement matrix, the motion matrix M representing the motion of the camera, and the shape matrix S representing the spatial position of the feature point are

[Equation 3] The relationship is established.

[0052] M is arranged basis vector _i f of the coordinate axes of the f-th frame of the camera, from the top of the transpose of the _{j f} in order 2F
A × 3 matrix, S is a 3 × P matrix in which three-dimensional coordinates with the centroids of p feature points as the origins are arranged in order from the left, and both ranks are 3 or less. Using this and singular value decomposition, W
When W is given, W is decomposed as in equation (3), and M,
The method of obtaining S is the above-mentioned factorization method of Tomasi and Kinade.

In this factorization method, since the images to be processed are handled together, they are not suitable for situations where new images are input one after another. However, for batch processing, it is a very excellent method that can restore the orientation of the target object and the three-dimensional shape of the feature points without obtaining camera parameters.

3. Method of Registering Shape Matrix Next, a method of obtaining and registering a shape matrix of feature points by using this factorization method in batch processing will be described with reference to FIG. This shape matrix is registered in the shape matrix registration unit 3 for each user.

FIG. 3 shows an example of the flow of shape registration processing.

First, the number of registered frames is initialized. That is, f = 0 is set (step 7).

Next, in order to photograph various face-oriented images of the user who wants to register, the pointer is displayed at the position on the display where the user wants his or her face to face.
To display the user's attention (st
ep8).

When the user turns his / her face in the direction of the pointer, the CCD camera of the image input section 1 is used to take an image (step
9).

The characteristic point detecting section 2 detects P characteristic points from the input image (step 10).

By repeating this, an image of the number of frames f required to obtain the shape is photographed (step 13).

The feature point detection unit 2 obtains the measurement matrix W from P (P> = 4) feature points corresponding to the image of the frame number f. Then, the shape matrix registration unit 3 applies the factorization method to the measurement matrix W to obtain the motion matrix M and the shape matrix S, and registers S (step 14).

This shape matrix S is unique to the user who wants to register. Therefore, when there are a plurality of users, the shape matrix is registered for each user.

The shape matrix S needs to be created based on a plurality of frame numbers f (f> = 3) as described above. This is because it is necessary to calculate the equation (3).

A character may be displayed on the pointer of step 8 or the direction can be notified by voice.

If the result of feature point detection is displayed in step 10, and if the detection result is not correct, the frame is not used, and the detection result is confirmed, it is possible to reliably obtain a highly accurate shape matrix.

Further, it is also possible to move the pointer and photograph a required number of frames while the user follows the pointer, and perform feature point detection and shape restoration collectively after the photographing.

4. Face Orientation Calculation Method Next, a calculation method for obtaining the orientation in real time when the image to be processed is input using the registered shape matrix will be described.

First, the image input unit 1 inputs an image for which the face orientation is desired.

Next, in the same way as described above, the feature point detection unit 2 performs P
Individual feature points are obtained, and the measurement matrix W _{new is calculated} from the feature points.
Ask for.

Expression (3) expresses the relationship between the coordinates of the characteristic points on the image, the spatial position thereof, and the camera axis in F frames. This holds even for only one frame.

Then, a new image for which the face orientation is desired
W is the measurement matrix and motion matrix of the image_new , M_newTable
However, since the 3D shape of the user's facial feature points is
The shape matrix S registered in is used.

The shape matrix S corresponding to the user is selected from the shape matrices registered in the shape matrix registration unit 3.
Call.

The input image in this face orientation calculation method, that is, the number of frames may be 1, and the measurement matrix W _new obtained from the feature points of this one frame is used. This requires three or more frame numbers f in the above-described shape matrix registration method, but in the face orientation calculation method,
This is because it is not necessary to calculate the formula (3), and it is sufficient to calculate the formula (8).

However, the number of feature points obtained from one new image must be the same as the number P of feature points used when creating the previously registered shape matrix.
If the numbers are different, it will be described later.

Now, the measurement matrix W _new and the motion matrix are M
_new and a user-specific shape matrix S registered in advance
Between

[Equation 4] The relationship is established.

The document “Regression and t”
he Moore-Penrosepseudoinv
erse (Albert, A. Academic
Press. 1972) ”, the matrix equation

[Equation 5] The general solution X of is

[Equation 6] Given in. Where A ⁺ is the general inverse of the matrix A,
Y represents an arbitrary matrix. Applying this to equation (4),
The motion matrix for the new frame is

[Equation 7] Given in. Here, SS ⁺ is a projection matrix of S to the range R (S). R (S) is a space spanned by a vertical vector of S, that is, a vector representing the three-dimensional coordinates of each feature point.

If at least four feature points out of P feature points are not on the same plane and are three-dimensional, R
The dimension of (S) is three-dimensional, and coincides with the entire space. At this time, SS ⁺ = I, and the equation (7) becomes

[Equation 8] Becomes

That is, the shape matrix S registered in advance
The generalized inverse matrix of can be used to find the motion matrix for the new frame.

The general inverse matrix is S⁺= S^T(SS^T)
⁺Given by R (SS^T) Is the same as the previous discussion
Since there is an inversion matrix, the inverse matrix exists, and in this case general
The inverse matrix is S⁺= S^T(SS^T)^-1Can be calculated by
You can Where S^T Represents the transposed matrix of S.

Once the general inverse matrix of the shape matrix is calculated once, the motion matrix can be calculated only by a very light process of multiplying the general inverse matrix by the measurement matrix obtained from sequentially input images.

Motion matrix M_newFixed the target object
Camera coordinate axes when the camera orientation is considered to have changed
Represents exercise. That is, the motion matrix M_newRow vector
Transpose of two of the three basis vectors of camera coordinates
Basis vector i_new, J _newAnd i
_new, J_newThe remaining basis vector k from the cross product of
_newIs required. When the camera is placed in front of the target object
If the camera is fixed based on the combined camera orientation
Rotation matrix R of the target object of_newIs i_new , J
_new, K_newIs obtained as a matrix in which
The rotation angle of the target object can be calculated from the rotation matrix
It

The unit vector representing the orientation of the target object (in the case of considering the face as the target object, a normal vector that looks like the face as a plane and goes outward from the face plane), the vector d is
The transposition of the third row of the row vector of R _{n ew} obtained by -1 times.

In summary, the x component of the normal vector d is
A component of the first row third column of motion matrix _{M new new,} y components of the normal vector d is -1 times the second row third column components of movement matrix _{M new new.}

FIG. 4 shows how a motion matrix M _new is calculated by considering a face as a target object.

In FIG. 4, the manner in which the camera is fixed and the face is moved is considered to relatively change the camera orientation with the face fixed.

FIG. 2 shows an example of the flow of face orientation calculation processing.

First, the general inverse matrix S ⁺ is obtained from the registered shape matrix and stored in the general inverse matrix storage unit 4 (step 1).

One frame of the user's face image is input by the image input unit 1 (step 2).

The feature point detection unit 2 detects feature points and calculates the measurement matrix W _new (step 3).

The face orientation estimation unit 5 calculates the product of the measurement matrix W _new and the saved S ⁺ as in the equation (8), and the orientation M _new in the frame is obtained (step 4).

By repeating step 2 to step 4, the face orientation of the user can be calculated by online processing.

The shape matrix is registered in the registration process, and the general inverse matrix is obtained in the face orientation calculation process. However, if the general inverse matrix is calculated in the registration process, the general inverse matrix in the face orientation calculation process is obtained. No need to calculate.

(Modification Example of Face Orientation Calculation Method) By the way, the number of feature points obtained from one new image is the same as the number P of feature points used when creating the pre-registered shape matrix. Must. However, the same feature point is not always displayed, and the registered feature point may be hidden depending on the orientation, or conversely, a new feature point that is not registered may be found.

In the face orientation calculation method of this embodiment,
It is possible to easily deal with the increase and decrease of such feature points.

When the characteristic point is hidden, the direction can be calculated by the equation (8) as usual by removing the point from the shape matrix.

When a new feature point is found, the motion matrix is calculated from the registered feature points, and the shape of the feature point is calculated from the product of the general inverse matrix of the obtained motion matrix and the measurement matrix of the new feature point. The matrix is obtained. If a new shape matrix is obtained by arranging these in a registered shape matrix, the feature points can be used from the next frame.

However, when the number of feature points increases or decreases in this way, it is necessary to obtain the measurement matrix using the centroids of the already-registered feature points, and therefore the centroids of the feature points must also be registered.

In the shape matrix registration method and face orientation calculation method of this embodiment, a face is described as an object, but the object is not limited to a face, and the orientation can be obtained for any object. .

(Mouth edge detection method) In the case of using the feature point-based method such as the case of using the factorization method in the registration of a three-dimensional shape or the case of calculating the face orientation based on the registered shape matrix, , It is necessary to accurately obtain the coordinates of the feature points of the face in the image.

The mouth end detection method will be described below with reference to FIG.

FIG. 5 shows an image of the mouth edge position and the area around the mouth.

The nature of the image around the mouth is that the part where the upper and lower lips meet is a continuous edge, and that there is little texture around the mouth.

First, edges are emphasized by applying a first-order differential filter or the like to the image around the mouth. If the method of specifying the object boundary (Japanese Patent Application No. 2001-173008) is used for the edge image, it is possible to perform the mouth edge detection utilizing the above two points. This method of identifying the object boundary is such that, when the target object is bright and the surroundings are dark, or vice versa, the difference between the brightness of the target pixel and the average brightness value of the pixels outside the target pixel is maximum at the boundary of the object. Therefore, it is a method of searching the pixel having the maximum value as the boundary of the object area.

Since the search area at the mouth end can be narrowed down by detecting the pupil and nostril using the Fukui-Yamaguchi separation degree filter described in the prior art, the above method can be applied to the search area. If applied, the mouth end can be detected stably.

(Pointer Display Automatic Adjustment Method) The above is the technique necessary for calculating the face orientation.
A case where the position where the pointer is displayed on the display is calculated from the calculation result of the face orientation will be described.

The pointer display position on the display is
It is obtained by imitating the face as a plane and projecting a normal vector md directed outward from the face plane on the display.

[0107]

[Equation 9] Here, the vector d is a unit vector indicating the direction of the normal line extending outward from the face plane, and can be easily obtained from the motion matrix as described above, for example.

M is a constant indicating a display scale determined by the range of the face direction moved by the user and the size of the display. If the value of m is set large, the face can be moved to the edge of the display at a small angle, and if the value of m is set small, the face must be moved at a large angle to move to the edge of the display. Which one is easier to use depends not only on the user but also on the size of the display. Therefore, it is necessary to obtain m for each user or display so as to match the range of the face of the user with the range of movement of the pointer.

First, an automatic pointer display adjustment method will be described.

This is a method of automatically adjusting the moving range of the pointer by using the statistics of the face orientation.

From the face orientation result obtained by the face orientation calculation, for example, the maximum and minimum values of the horizontal angle of the display and the maximum and minimum values of the vertical angle are updated, and a certain time At the point in time, the display scale of the pointer position m
Should be changed.

If the case of facing the outside of the display is taken into consideration, the distribution of angles is taken, and instead of the maximum value and the minimum value, the maximum value and the minimum value of the area including 40% to the left and right from the intermediate value are used. You can also

By using the face orientation statistics in this way, the face orientation range can be narrowed down and the pointer display position can be obtained by making the range correspond to the size of the display. It is possible to prevent deviation from the moving range, and it becomes easy to use in conjunction with intuition.

(Pointer Display User Adjustment Method) A pointer display adjustment method by the user will be described.

By allowing the user to input the value of the display scale m, the display scale can be set for the user's convenience while actually trying various values. Also, based on what was roughly automatically adjusted by the above-mentioned automatic adjustment method, by this user adjustment method,
The user can also make fine adjustments.

As described above, according to the present invention, it is possible to realize the pointer device 20 utilizing the face direction.

If the face-pointing pointer display device 20 realized by this embodiment is used for cursor control of a PC, P
The work efficiency of the C operation can be improved.

(Modification) In addition, it can be used as an input device for home electric appliances, for example.

Further, it can be used as a character input device in which the display is divided into several areas to display characters and the area where the face is directed can be selected.

If the face orientation calculation technology is considered alone, the face orientation calculation technology can be applied not only to pointer devices but also to drivers of vehicles, spectators of exhibits, operators of game terminals, and the like.

For the driver of the vehicle, it is possible to detect an inattentiveness that interferes with driving by the face orientation calculation and sound an alarm, or to present appropriate information for safe operation according to the face orientation. . For the spectators of the exhibit, it is possible to identify the exhibit that the spectator is paying attention to by face-to-face calculation and automatically present information regarding the exhibit.

For the operator of the game terminal, the viewpoint of the game environment can be changed by face orientation calculation, the aim of shooting can be set, and the swinging motion can be used for communication with the characters. it can. As described above, the technique relating to the face-pointing pointer can be used in various fields.

[0123]

As described above, according to the present invention, a three-dimensional shape of a feature point of an object such as a face is accurately obtained and registered in advance, and a general inverse matrix of the registered shape matrix is used. By doing so, the face orientation can be calculated at high speed and accurately from the images input one after another.

From the calculation result, the pointer can be displayed at the position on the display where the face is facing.

[Brief description of drawings]

FIG. 1 illustrates a mechanism of a pointer display device that uses face orientation. The dotted arrow represents the flow of shape registration processing, and the solid arrow represents the flow of face orientation calculation processing.

FIG. 2 shows a flow of face orientation calculation processing.

FIG. 3 shows a flow of shape registration processing.

FIG. 4 shows a face orientation calculation method.

FIG. 5 shows images of a mouth end position and a mouth area.

[Explanation of symbols]

1 Image input section 2 Feature point detector 3 Shape matrix registration section 4 General inverse matrix storage 5 Face orientation calculator 6 pointer display

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Jun Maki             1st Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa             Inside the Toshiba Research and Development Center F term (reference) 5B087 AA09 AA10 DD09 DE07                 5L096 CA04 DA02 FA09 HA09 JA11                       KA15

Claims (7)

[Claims] 1. An object orientation calculation apparatus for calculating an orientation of an object, wherein a shape matrix representing a spatial position of a feature point of the object is created in advance, and a general inverse matrix is obtained from the shape matrix. Calculating means, image input means for inputting an image in which the object is captured, feature point detecting means for detecting feature points of the object from the image, and creating a measurement matrix from these feature points, the feature An object orientation calculation means for calculating a motion matrix expressing the orientation information of the object by using a measurement matrix created by the point detection means and a general inverse matrix for the object created in advance, Orientation calculator. 2. The general inverse matrix calculating means inputs a plurality of images in which various directions of the object are photographed by the image inputting means, and the characteristic point detecting means calculates characteristic points of the object from the respective images. And each of them is detected, one measurement matrix peculiar to the object is created from these feature points, the shape matrix peculiar to the object is obtained from this measurement matrix using the factorization method, and the general inverse matrix is obtained from this shape matrix. The object orientation calculation apparatus according to claim 1, wherein: 3. A pointer display in which a point on the display on which the human face is facing is calculated from a motion matrix representing the human face facing information calculated by the object orientation calculating means, and a pointer is displayed on the point. The object orientation calculation apparatus according to claim 1, further comprising means. 4. The pointer display means obtains a face normal vector indicating the face orientation from the motion matrix calculated by the object orientation calculation means, adjusts the coefficient of this normal vector, and moves the pointer. The object orientation calculation apparatus according to claim 1, wherein 5. A feature point detecting device for detecting a feature point of a face, wherein the mouth end is represented based on a continuous edge of the mouth and a brightness around the mouth whose brightness is brighter than the edge of the mouth. A feature point detection device characterized by detecting feature points. 6. An object orientation calculation method for calculating an orientation of an object, comprising: creating a shape matrix representing a spatial position of a feature point of the object in advance; and obtaining a general inverse matrix from the shape matrix. A calculation step; an image input step of inputting an image of the object captured; a feature point detection step of detecting feature points of the object from the image and creating a measurement matrix from these feature points; An object orientation calculation step of calculating a motion matrix representing the orientation information of the object using a measurement matrix created in the point detection step and a general inverse matrix created in advance for the object, Direction calculation method. 7. A program for realizing a method for calculating an orientation of an object by a computer, the shape matrix representing a spatial position of a feature point of the object is created in advance, and a general inverse from the shape matrix. A general inverse matrix calculation function for obtaining a matrix, an image input function for inputting an image in which the object is photographed, a feature check for detecting feature points of the object from the image and creating a measurement matrix from these feature points The output function, the measurement matrix created by the feature point detection function, and the object orientation calculation function that calculates a motion matrix representing the orientation information of the object using a general inverse matrix created in advance for the object are realized. A program for an object orientation calculation method characterized by: