JPH09146692A - Image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To move a position or an attitude to a desired position by freely moving it unless the face gets out of a measurement area by moving a pointer cursor to a target position found by a target pointer cursor position calculating means while using a pointer cursor moving means. SOLUTION: The direction of the face of the user operating a keyboard 6 while facing a display 5 is measured by a face azimuth measuring means 1 with six degrees of freedom, and the target position of pointer cursor on the screen of display 5 corresponding to the position and attitude of that measured face is calculated by a target pointer cursor position calculating means 2. Then, a pointer cursor moving means 3 moves the cursor displayed on the display 5 to the target position found by the calculation while controlling a computer 4. Thus, even when the position or attitude of face is freely moved, it can be moved to the desired position unless face gets out of the measurement area.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image display device having means for moving the position of a pointer cursor on a display screen.

[0002]

2. Description of the Related Art The current computer is Mac of Apple Inc.
An interface using a pointer cursor (hereinafter referred to as a cursor) and an icon is generally used as seen in Windows and Windows of Microsoft Corporation. Conventionally, as a device for moving the cursor, there are a mouse, a trackball, a joystick, a tablet, a touch panel, and the like. The mouse seems to be the best interface so far.

However, when developing a program in a window environment such as X window, it is necessary to remove the hand from the keyboard when switching windows.
I feel annoying. Further, even when the program is not being developed, for example, in the case of inputting text by a word processor, the operation of releasing the hand from the keyboard is troublesome, and the input efficiency is reduced. Trackballs and joysticks are used as interfaces that do not require you to remove your hands from the keyboard, and recently, pads that use touch panels (A
Some of them are installed on the keyboard or near the keyboard, but there is a problem in operability.

Several proposals have been made as described below as a cursor movement method which does not require a special device operation without releasing the hand from the keyboard. In Japanese Patent Laid-Open No. 62-143124, a sensor using a gyro is attached to the user's head to move the head in the left-right direction (which is considered to be the left-right rotation around the central axis of the head) and the movement in the up-down direction. There is a proposal to detect (a vertical movement in a vertical direction, but not described in detail in the above document) and move the cursor position by the movement.

In this method, since the sensor detects only the movement of the head up, down, left and right, it can be performed only under very limited conditions (the position of the user's face is always located in front of the display, etc.). , The cursor does not follow the direction the face is facing. In other words, if the user moves the head position only in translation without swinging the head vertically or horizontally, the cursor does not follow. Since the positional relationship between the head and the display varies depending on the person, such a method compels the user to position the head and the posture, which is a pain for the user, and the cursor is not intuitive to the user. It cannot be moved to the desired position that suits.

Further, Japanese Patent Laid-Open No. 59-117631 proposes a method that is substantially similar to the above-mentioned method, and only the sensor is different. In this proposal, a sensor using gravity is used to detect up and down, and a sensor using ultrasonic waves is used to detect left and right. This proposal is also basically based on the above-mentioned reference (Japanese Patent Laid-Open No. 62-62160)
Since it is the same as the method proposed in Japanese Laid-Open Patent Publication No. 143124), it has the same problem as described above.

Further, in Japanese Patent Laid-Open No. 3-176718, a light emitting element attached to the upper part of a display is captured by a video camera attached to the user's head, and the position of spot light emitted from the light emitting element is known by image processing. , A method of moving a cursor has been proposed. This proposal also requires the user's head position to be restrained in order to move the cursor accurately in the direction in which the face is facing, and the user cannot work in any position or posture.

The above proposal uses the swing of the user's head (up, down, left and right).
In Japanese Patent Laid-Open No. 226827 and Japanese Patent Laid-Open No. 2-212918, a method of detecting a user's line of sight and moving a pointer cursor in that direction is proposed. The former is to attach a line-of-sight detection sensor including a half mirror and a lens to a helmet, and the user covers it with the line-of-sight direction to detect the line-of-sight direction and move the pointer cursor in the line-of-sight direction. According to this proposal, the user is forced to wear a helmet with a line-of-sight detection sensor, which is not convenient. Also, the display is viewed through the half mirror, which may make it difficult to see the screen. In the latter, the line of sight is detected by detecting the movement of the user's black eye with a video camera attached to the upper part of the display, but there is no description of a specific processing method or the like, and even if the position of the black eye in the eye is Even if it can be detected, the direction of the face etc. will be related to the gaze direction,
It is completely unknown how to detect the line of sight.

[0009]

SUMMARY OF THE INVENTION In view of the above circumstances, the present invention occurs in a normal operation such as operating a keyboard while observing a display screen without forcing the user to position or position the head. It is possible to provide an image display device that allows a change in the obtained position and orientation and can move the pointer cursor on the display screen without operating an operator for moving the pointer cursor on the display screen. To aim.

[0010]

An image display device of the present invention which achieves the above object has a display screen for displaying an image,
On the display screen, display means for displaying an image including a pointer cursor for pointing a point on the display screen,
In an image display device having pointer cursor moving means for moving the position of the pointer cursor displayed on the display screen, 6-degree-of-freedom face orientation measuring means for measuring the position and posture of the face facing the display screen, A pointer cursor target position calculating means for calculating a target position of the pointer cursor on the display screen according to the position and orientation of the face measured by the 6-degree-of-freedom face orientation measuring means, and the pointer cursor moving means, It is characterized in that the pointer cursor is moved to the target position obtained by the pointer cursor target position calculation means.

Since the image display device of the present invention is provided with the 6-degree-of-freedom measuring means for measuring the position and orientation of the face facing the display screen, that is, the 6-degree-of-freedom, as long as the face does not deviate from the measurement area, the position It is possible to move the pointer cursor on the display screen in the direction in which the face is directed, that is, to a desired position adapted to intuition, even if the posture is freely moved.

Here, in the image display device of the present invention, the 6-degree-of-freedom face orientation measuring means has a target mark fixed on the face side, in which the shape when observed from a predetermined direction changes depending on the posture. And image pickup means for obtaining image data representing an image when the target mark is viewed from a predetermined direction, and face orientation calculation for calculating the position and orientation of the face based on the image data. It may be provided with a means.

In the case of this method, it is necessary to fix the target mark on the face side, for example, the spectacles or the head, but the target mark may be such that the position and posture of the face can be accurately measured. Therefore, the weight is light and the user is not bothered. on the other hand,
By using such a target mark, the position and posture of the face can be accurately measured.

Alternatively, in the image display device of the present invention, the 6-degree-of-freedom face orientation measuring means obtains image data representing an image when the face is viewed from a predetermined direction by photographing the face. A feature point tracking means for tracking the positions on the image of at least four feature points appearing on the image obtained by the image pickup means based on the image data; It may be provided with a face orientation calculating means for calculating the position and orientation of the face based on the positions of at least four feature points on the image.

In the case of this system, it is not necessary to wear anything including the target mark on the face or the head, and the user is completely free from the trouble. Here, it is preferable that the image display device of the present invention includes a pointer cursor variation suppressing unit that suppresses variation in the display position of the pointer cursor. If the position of the pointer cursor changes sensitively to changes in the position or posture of the face or head, the screen may be difficult to see. In addition, there is a possibility that the position variation (vibration) of the pointer cursor may occur due to the measurement resolution and measurement error of the position and orientation of the face.
Therefore, a pointer cursor variation control means is provided so that the user does not intend to move the position of the pointer cursor,
Suppressing the position change of the pointer cursor so that the position change of the face or head or the position change of the pointer cursor caused by the measurement error of the position or position of the face or the like is prevented. , The visibility of the screen is improved.

In the image display device of the present invention, an operator for inputting whether or not to move the pointer cursor in response to the position and orientation of the face measured by the 6-degree-of-freedom face orientation measuring means. It is preferable to provide. In this case, a keyboard provided with a plurality of operation keys may be provided, and at least one operation key among the plurality of operation keys arranged on the keyboard may be the operator.

In the actual work, the pointer cursor may or may not need to be moved depending on the work, or may not be moved. The provision of the above-mentioned operator realizes an image display device which is suitable for a plurality of such cases and is further improved in usability.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below. 1 to 3 are block diagrams showing a schematic configuration of each embodiment of the image forming apparatus of the present invention. FIG. 1 is a block diagram showing a basic embodiment of the present invention. As shown in FIG.
The orientation (position and orientation) of the face of the user who is operating the keyboard 6 facing the display 5 is measured, and the pointer cursor target position calculation means 2 determines the movement target position of the cursor based on the orientation of the user's face. And the pointer cursor moving means 3 controls the calculator 4 to move the cursor displayed on the display 5 to the target position calculated.

FIG. 2 shows a structure provided with a cursor fluctuation suppressing means 7 for suppressing unnecessary fluctuation of the cursor. Further, FIG. 3 is an example provided with an operator 8 for instructing whether or not to move the cursor according to the direction of the face of the user. FIG. 4 is an image diagram when the present invention is applied to a computer system.
Shown in According to each embodiment of the present invention, as shown in FIG. 4, the cursor can be made to accurately follow the direction in which the user's face is facing.

Each means in the embodiments shown in FIGS. 1 to 3 will be described in detail below. First, a method for obtaining the cursor movement target position on the screen of the display 5 from the direction of the user's face will be described. FIG. 5 shows the relationship between the face coordinate form and the screen coordinate form. In order to obtain the cursor position from the face direction, the cursor may be moved to the direction where the user's face is facing. Specifically, at the point where a straight line whose direction vector is a vector that is perpendicular to the plane of the face (which is not actually a plane but approximates a plane) intersects the plane of the display (called the screen plane). Move the cursor. Here, the orientation of the face (6 degrees of freedom (x, y, z,
A method for obtaining the cursor position (2 degrees of freedom (x, y)) from (roll, pitch, yaw)) will be described. FIG. 6 shows the definitions of roll (roll), pitch (pitch), and yaw (yaw).

The vector Vsf from the origin of the screen coordinate form to the origin of the face coordinate form is Vsf = [x _f , y _f , z
_f , φ, θ, ψ] (posture is PRY (pitch, roll
l, yaw) expression), in matrix expression,

[0022]

(Equation 1)

Since the unit vector in the z-axis direction of the face coordinate form expressed in the face coordinate form is [0,0,1,0], the direction vector Vzfs of the face coordinate form expressed in the screen coordinate form in the z-axis direction. Is

[0024]

(Equation 2)

## EQU1 ## here,

[0026]

(Equation 3)

In other words, Vzfs = [a, b, c, 0]
Is a direction vector, and the equation of a straight line passing through the origin of the face coordinate form is

[0028]

(Equation 4)

Then, this straight line and x of the screen coordinate form
The point where the y-plane intersects is the position of the desired cursor P _c =
(X _c , y _c , 0). Therefore, by substituting P _c into this straight line equation and rearranging,

[0030]

(Equation 5)

From the equations (1) and (2), the position and orientation of the face coordinate form expressed in the screen coordinate form Vsf = [x _f ,
The position (x _c , y _c ) where the cursor is moved can be calculated from y _f , z _f , φ, θ, ψ]. An embodiment using a target mark is shown in FIG. As shown in the figure, a target mark is attached to a part of the user's face (head in the figure), and the target mark is captured by the camera attached to the top of the display, and the visual tracking system installed on the visual processing board is used. Real-time measurement of the user's face orientation (position and posture). The monitor is for displaying an image taken by the camera. The visual tracking system used here is, for example, “Japanese Patent Application (International Publication No. W093) published internationally under the Patent Cooperation Treaty.
/ 15376) ”. The method of measuring the target mark using this system is shown below.

FIG. 8 shows the target mark used here. As shown in the figure, place a white triangle on top of the black circle with a certain amount of shift. This target mark is for detecting the deviation of the posture by utilizing the fact that the positional relationship between the triangle and the circle changes as shown in FIG. 9 when the viewpoint for viewing the target mark changes. The amount of deviation between the triangle and the circle can be easily obtained from the projection histogram.

The target mark is detected by running a window of a certain size on the screen and examining the projection histogram in that window. Figure 1 shows this situation.
0 is shown. In the case of a target mark, whether or not it is a target mark can be determined by having two peaks in the projection histogram for both x and y. FIG. 11 shows how to obtain the deviations in the x and y directions. As shown in this figure, the difference (1x, 1y) between the center position of the target mark and the preset target position is xy on the image coordinate form.
Deviation. Since what is actually required is a deviation on the camera coordinate form, this value (lx, ly) is converted to the camera coordinate form. The coordinates of the center of the target mark are also obtained from the xy projection histogram.

As shown in FIG. 11, the deviation in the z direction is obtained by the difference between the measured mark area corresponding to the target z value of the target mark and the measured area. Next, how to obtain the deviation in the posture (roll, pitch, yaw) direction will be shown.
First, how to find the deviation between the pitch and yaw directions will be described. These are obtained from the positional relationship between the lower black circle and the upper white triangle forming the target mark. When the camera's viewpoint is tilted, the triangular mark appears to be offset from the center of the circle. This deviation is the deviation in the pitch and yaw directions. The deviation between the triangle and the circle is also calculated from the projection histogram. This method is shown in FIG. First, the peak point is obtained from the x projection histogram (Xp1, Xp2 in FIG. 12). This Xp1 and Xp2
The part between is the triangular part. It suffices to find which part of the entire mark the triangular part is located. That is,
Difference L between the center Xbc of the entire mark and the center Xpc of the triangle
yaw is the deviation in the yaw direction. Similarly, the deviation Lpitch in the pitch direction is obtained from the y projection histogram.

Finally, a method of obtaining the deviation in the roll direction will be shown. This is shown in FIG. The deviation in the roll direction is x of one vertex of the triangle (A in FIG. 14) due to the rotation in the roll direction.
The position shifts from the center line of the triangle. The deviation in the roll direction is also obtained from the x projection histogram. That is, as shown in FIG. 14, the midpoint Xp of Xp1 and Xp2
The difference between c and X1p is the deviation Lroll in the desired roll direction.
l.

The deviation obtained as described above is represented by the number of pixels. This is converted into an absolute scale (position is millimeter, posture is radian) by a conversion function obtained by actual measurement. The method of obtaining this conversion function is shown below. First, as shown in FIG. 15, for each point in the z-axis direction of the coordinate shape fixed to the target mark, first, the value of the z-axis (distance between the camera and the target mark) and the screen of the target mark at that time. The deviation Lz (the number of pixels) above is measured. Next, the positive and negative points are moved in the x and y axis directions, and the movement amounts Dx and Dy at that time and the deviations Lx and Ly (the number of pixels) of the mark on the screen are measured. further,
The camera is rotated by several points around the x, y, z axes of the coordinate shape of the target mark, and the movement amounts Dyaw, Dpit at that time
The deviation between ch, Drroll and the mark on the screen is measured. The above measurement is performed at each point in the z-axis direction.

From the above measurement, the following relationship is obtained. (1) Distance between camera and target mark (approx. Z)
And the area of the target mark on the screen. (2) Each axial direction (x, y, z, roll, pitch, ya) when the distance between the camera and the target mark is constant.
The relationship between the deviation in the actual coordinates of w) and the deviation on the screen. (3) Behavior of the relationship of (2) with respect to the distance between the camera and the target mark.

FIG. 16 is a graph showing these relationships. As shown in FIG. 16 (b), the relationship of (2) can be approximated by a straight line passing through the origin, and the inclination of this straight line changes according to z as shown in FIG. 16 (c). Although this figure shows the behavior of y, the same behavior is exhibited for each of the other axes (excluding z), so that the approximate expression A for each axis direction is obtained.
_* = G _* (z) is obtained.

From the above results, the deviation L _* (pixels) obtained by the image processing can be converted into the deviation D _* (millimeters, radians) in the actual coordinates by the procedure shown in FIG.
FIG. 18 is an example in which target marks are attached to eyeglasses that protect the eyes from the display. By doing so, the user does not have to be conscious of wearing the protective glasses and conscious of putting the target mark, so that the user is not bothered. FIG. 19 shows an example in which the headset is marked. Also in this case, there is no annoyance of attaching the target mark.

FIG. 20 shows an example provided with a feature point tracking means for tracking face feature points. Here, for example, the facial feature points as shown in FIG. 21 are tracked in real time according to the algorithm incorporated in the feature point tracking board shown in FIG. 20, and the positions of those feature points on the image are obtained. In this example, the feature points are the outer corners of the eyes and the left and right edges of the mouth.
These feature points may be any part of the face as long as the mutual positional relationship of the two feature points is known. A method of calculating the orientation of the face from the positions of the feature points on the image can be obtained by solving the 4-point perspective problem if the face is approximated as a plane.

Although the algorithm for the four-point perspective problem is known, it will be described below. (Coordinate system) The coordinate system is set as shown in FIG. Set the reference coordinate system on the screen. At this time, the origin O is set at a position where the optical axis and the screen intersect, and a right-handed system is set in which the line-of-sight direction is the Z-axis, the left side toward the line of sight is the X-axis, and the upper side is the Y-axis. Further, the focus is assumed to exist at a position separated by f in the negative direction of the Z axis, and modeling is performed. For the actual perspective transformation matrix, it is necessary to consider the actual lens system, but here, a simple modeling is performed.

As the coordinate system of the object (user's face),
One of the four feature points used for measurement is set as the object coordinate origin O '. Let Z ′ be the normal direction of the plane where the four points exist.
With respect to the X'axis and the Y'axis, the coordinate system is taken as a right-handed system. In the coordinate system O'constructed in this way, three vectors of O'and the other three feature points are A (a1, a2, 0), B (b1, b2, 0),
Let C (c1, c2, 0) (each point lies in the X'Y 'plane).

The projection point of each feature point on the image plane is
The vector from the origin O to the projection point of the object coordinate origin is
O1 (o1, o2, 0) and vectors to the projection points of each feature point are α (α1, α2, 0), β (β1, β
2, 0) and γ (γ1, γ2, 0). From the above,
It is possible to represent each feature point and the feature point projected on the screen with respect to each coordinate system. (Coordinate conversion) First, the vector A from the object coordinate origin,
A transformation matrix that connects B and C with the feature point vectors α, β, and γ on the image plane is obtained. Hereinafter, the vector is represented by the simultaneous coordinate display in which the fourth component is standardized to 1.

As can be seen from FIG. 22, the coordinate system O'-X '
It can be understood that Y′Z ′ is obtained by translating the coordinate system O-XYZ by s in the X-axis direction, t in the Y-axis direction, and u in the Z-axis direction, and further rotating it. Therefore, the vectors A, B, C
And the vectors α, β, and γ are linked by a matrix formed by the product of a transformation matrix T representing translation / rotation and a transformation matrix representing the perspective transformation PzP. U is a unit conversion matrix, and p =
f / F (f: focal length represented by the number of pixels, F: focal length represented by a unit such as meters) is the number of pixels per unit length.

That is, X = P _z PTU · X ′ (3) Here, X ′ is a vector represented by the O′-X′Y′Z ′ system, and X is X. It is a diagram in which a vector that has been perspective-transformed on the −Y plane is displayed in an O-XYZ system. Also, P _z ,
P, T, and U are the following matrices, respectively.

[0046]

(Equation 6)

Now, T is an O '= X'Y'Z' system and O
-Represents the translation / rotation between XYZ systems,

[0048]

(Equation 7)

Equations (8) to (10) are respectively O'-
The unit vectors of the X'axis, Y'axis, and Z'axis of the X'Y'Z 'system are represented in the O-XYZ system. P is the perspective transformation of the focal length f, and P _z is the projection transformation on the XY plane. Here, when P _z PTU is calculated, the following conversion formula is finally obtained.

[0050]

(Equation 8)

(Component relational expression) In the expression (11), X '
Substitute A, B, and C for X, and α, β, and γ for X. That is,

[0052]

(Equation 9)

Of these, (11), (13), (1
From equation 6),

[0054]

(Equation 10)

Therefore, in summary,

[0056]

[Equation 11]

Since equation (20) is a simultaneous coordinate display,
It can be modified as follows.

[0058]

(Equation 12)

Here, the projection point O1 of the object coordinate origin O '
Can be expressed as follows:

[0060]

(Equation 13)

Comparing both sides of the equation (21), (23),
Substituting equation (24) and rearranging,

[0062]

[Equation 14]

Is obtained. Where (11), (14),
The equations (17) and (11), (15), and (18) can be similarly rearranged as follows.

[0064]

(Equation 15)

By rearranging the above equations, the following linear equation is obtained.

[0066]

(Equation 16)

Here, the matrix M of 6 × 6 in the equation (31) is called. Therefore, the following is obtained from the equation (31).

[0068]

[Equation 17]

Here, n _{x '} = (n ₁₁ , n ₂₁ , n ₃₁ ) ^T ,
n _{Y ′} = (n ₁₂ , n ₂₂ , n ₃₂ ) ^T can be obtained from the equation (32). Further, s, t, u are obtained as follows. Norm of n _{x 'First,} since a unit vector,

[0070]

(Equation 18)

From the equation (32), Q is derived and becomes the following.

[0072]

[Equation 19]

Therefore

[0074]

(Equation 20)

From equations (23), (24) and (35),

[0076]

(Equation 21)

It becomes (O ', A, B, C spatial arrangement) (32), (36),
From equations (37) and (38), the coordinate system O'-X'Y'Z '
The conversion equations between the coordinate system and the coordinate system O-XYZ are all determined and can be expressed by the following equations.

[0078]

(Equation 22)

Here, A is called a transformation matrix, and A
Is

[0080]

(Equation 23)

Is given by It is assumed that X ′ has been converted into a pixel value. (Distance / Posture Representation Method) The position / posture of the object can be represented by the conversion matrix A derived by performing the formulation as described above. However, such a matrix expression format cannot particularly intuitively indicate the posture of the object to the operator. Therefore, in the following, roll-pitc
A derivation method from the conversion matrix A for the h-yaw expression format will be shown. (Roll-pitch-yaw representation format) The conversion matrix A is shown in equation (40).

[0082]

(Equation 24)

Here, n _{11 to} n in the conversion matrix
₃₃ represents the rotation component to be obtained. Transformation matrix A
Using, the rotation angle around the Z axis, Y axis, and X axis (roll,
Pitch, yaw angle, hereinafter denoted by RPY (φ, θ, ψ). ). Now, the rotation matrix of the X axis, Y axis, and Z axis is expressed as follows.

[0084]

(Equation 25)

Roll-pitch-yaw is a rotation ψ about the x axis of the original coordinate system, then a rotation θ about the y axis,
Finally, rotation φ around the z axis is performed to express the rotation. From this, RPY (φ, θ, ψ) is as follows using the expressions of (41), (42), and (43).

[0086]

(Equation 26)

RPY (φ, θ, ψ) is (40)
Since it represents the rotation component of the equation,

[0088]

[Equation 27]

## EQU11 ## Therefore, from equations (44) and (45),

[0090]

[Equation 28]

Then,

[0092]

(Equation 29)

Is obtained. Here, (44) can be modified as follows.

[0094]

[Equation 30]

The left side of the equation (50) is n _{11 to} n ₃₃ and (4
It is known from the result of φ in the equation 8).

[0096]

(Equation 31)

(1,1) and (3, on the left side of the equation (50).
From the 1) element and the equation (51),

[0098]

(Equation 32)

Then,

[0100]

[Equation 33]

As described above, the angular rotation angles roll, pitch, yaw angle can be obtained from the equations (48), (49) and (53). This is the end of the description of the algorithm for the four-point perspective problem. In both the method using the target mark and the method for tracking the facial feature point described above, the position / orientation of the target mark (or the position / orientation of the origin of the facial feature point) is measured in the camera coordinate form. , It is necessary to convert it to screen coordinates. Further, the coordinate form of the mark (the coordinate system of the origin of the face feature point) is also deviated from the face coordinate form, so that conversion is required.
In order to use the equations (1) and (2) already shown, the matrix Msf representing the position and orientation of the face coordinate form represented by the screen coordinate form may be obtained. FIG. 23 shows the relationship between the mark coordinate form (face feature point origin coordinate form) and the camera coordinate form. The following is a generalization by adding the mark coordinate form (face feature point origin coordinate form) and the camera coordinate form.

When Msc, Mcm, and Mmf are determined as shown in FIG. 23, the matrix Msf showing the position and orientation of the face coordinate type expressed in the screen coordinate type desired to be calculated can be calculated by the following equation. Msf = Msc · Mcm · Mmf (54) Here, Msc is a known position because the camera is installed. Further, Mmf is known because it is the positional relationship between the mark and the face. Mcm is obtained by visual measurement. Therefore, Msf can be calculated from these, and the cursor position is obtained from Msf thus obtained using the above-mentioned (1) and (2). Thus, using equation (54),
The positions and postures of the mark coordinate form (face feature point coordinate form) and the camera coordinate form can be arbitrarily determined.

When the resolution of visual measurement is insufficient,
When noise causes an error in measurement, it is expected that the cursor target position calculated from such data will be affected and the cursor will fluctuate finely. In such a case, a configuration example of the filter that suppresses the fluctuation of the cursor is shown below.

[0104]

(Equation 34)

This expression averages some past cursor command values (the number is determined by k) and weighted (determined by m) the present cursor command values to diffuse the influence of noise. When m is set to be large, the degree of noise diffusion increases and the influence of noise decreases, but the responsiveness deteriorates.
If such a filter is used, it is possible to reduce small fluctuations of the cursor, but it is not possible to stop it completely. Further, it is desirable that the cursor be stationary when it is not necessary (when the user does not intend to move the cursor). This problem can be solved by providing a switch on the keyboard that the user turns on when needed. As shown in FIG. 24, the state of the switch (ON, OFF) is read by PIO (parallel input / output interface), and the cursor is moved only when the switch is in the ON state (when the bit of PIO is 1). FIG. 25 is an example of a keyboard in which this switch is arranged below the space key. Keys other than the space key are not shown. In this way, if you place it under the space key, you can operate the switch with your thumb, which is less frequently operated.
It is efficient. Further, in this example, the switch is physically provided, but this function may be assigned to a key already existing on the keyboard. For example, control-x or shi
Like ft-space.

[0106]

As described above, since the image display device of the present invention measures the position and posture of the face facing the display screen, it allows the user to freely move the face or head. Moreover, the pointer cursor can be moved to the position where the face is facing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of an embodiment of an image forming apparatus of the present invention.

FIG. 2 is a block diagram showing a schematic configuration of another embodiment of the image forming apparatus of the invention.

FIG. 3 is a block diagram showing a schematic configuration of another embodiment of the image forming apparatus of the present invention.

FIG. 4 is an image diagram when the present invention is applied to a computer system.

FIG. 5 is a diagram showing a relationship between a face coordinate form and a screen coordinate form.

FIG. 6 is a diagram showing definitions of roll, pitch, and yaw.

FIG. 7 is a diagram showing an embodiment using a target mark.

FIG. 8 is a diagram showing a structure of a target mark.

FIG. 9 is a diagram showing an example of how a target mark looks with respect to a change in viewpoint.

FIG. 10 is a schematic diagram showing a method of detecting a target mark.

FIG. 11 is a diagram showing a method of detecting deviations in x and y directions.

FIG. 12 is a diagram showing a method of detecting deviation in the z direction.

FIG. 13 is a diagram showing a method of detecting deviations in the yaw and pitch directions.

FIG. 14 is a diagram showing a method of detecting deviation in the roll direction.

FIG. 15 is an explanatory diagram of a deviation measuring method.

FIG. 16 is a diagram showing the correspondence between the position and orientation of the target mark and the measurement result obtained by the camera.

FIG. 17 is a diagram showing a flow of converting a deviation indicated by the number of pixels into a deviation in real coordinates.

FIG. 18 is a diagram showing an example in which target marks are attached to eyeglasses.

FIG. 19 is a diagram showing an example in which a target mark is attached to a headset.

FIG. 20 is a diagram showing an embodiment including feature point value trace means.

FIG. 21 is a schematic diagram showing an example of facial feature points.

FIG. 22 is a diagram showing a coordinate system in the 4-point perspective problem.

FIG. 23: Coordinate system, Face, Mark, Cam, Sc
It is a figure which shows the relationship of reen.

FIG. 24 is a diagram showing an embodiment including a switch that determines whether or not to move the cursor depending on the orientation of the face.

FIG. 25 is a diagram showing an example of a switch that determines whether or not to move the cursor depending on the orientation of the face.

Claims (6)

[Claims] 1. A display unit having a display screen for displaying an image, the display unit displaying an image including a pointer cursor for pointing a point on the display screen, and a display unit displayed on the display screen. In an image display device provided with a pointer cursor moving means for moving the position of a pointer cursor, a 6-degree-of-freedom face orientation measuring means for measuring the position and orientation of the face facing the display screen, and a 6-degree-of-freedom face orientation measuring means. A pointer cursor target position calculating means for calculating a target position of the pointer cursor on the display screen according to the position and posture of the face measured by the measuring means, wherein the pointer cursor moving means is the pointer cursor target position. An image display device, characterized in that a pointer cursor is moved to the target position obtained by a calculation means. 2. The shape of the 6-degree-of-freedom face orientation measuring means when observed from a predetermined direction changes depending on the posture.
A target mark fixed on the face side; an image pickup means for obtaining image data representing an image when the target mark is viewed from a predetermined direction by photographing the target mark; and the face based on the image data. The image display apparatus according to claim 1, further comprising a face orientation calculation unit that calculates the position and orientation of the image. 3. The 6-degree-of-freedom face orientation measuring means obtains image data representing an image when the face is viewed from a predetermined direction by photographing the face, and based on the image data, Feature point tracking means for tracking the positions of at least four feature points appearing on the image obtained by the image pickup means on the image, and an image of the at least four feature points obtained by the feature point tracking means The image display device according to claim 1, further comprising a face orientation calculation unit that calculates the position and orientation of the face based on the position above. 4. The image display device according to claim 1, further comprising pointer cursor variation suppressing means for suppressing variation in the display position of the pointer cursor. 5. A manipulator for inputting whether or not to move the pointer cursor in response to the position and orientation of the face measured by the 6-degree-of-freedom face orientation measuring means. The image display device according to claim 1. 6. A keyboard having a plurality of operation keys arranged therein, and at least one operation key of the plurality of operation keys arranged on the keyboard is the operator. Image display device.