JPH0546309A - Sight line recognizing device - Google Patents

Abstract

PURPOSE:To stably and exactly execute the sight line detection by eliminating an eyeball jump component of an eyeball displacement amount detection output, and allowing an eyeball displacement amount and a coordinate on a display screen to correspond exactly to each other. CONSTITUTION:The device is provided with an eyeball displacement amount detecting means 1 for detecting a displacement amount of an eyeball corresponding to a variation of an operator's sight line, a signal smoothing means 2 for smoothing a displacement amount detection output from the eyeball displacement amount detecting means 1, and a neural network 3 for learning and recognizing to allow an eyeball displacement amount smoothing output from the signal smoothing means 2 and a coordinate on a display screen to correspond to each other. Accordingly, an eyeball jump component from the eyeball displacement amount detecting means 1 is eliminated, and by leaning and recognizing non- linearity to the screen coordinate by using the neural network 3, the sight line can be detected stably and exactly.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight recognition apparatus for operating an information group displayed on a display means such as a CRT display by using the line of sight of an operator.

[0002]

2. Description of the Related Art Generally, in a workstation or other information processing system, selection of an icon displayed on a display means (hereinafter referred to as a display screen) such as a CRT as a user interface and instruction for moving a cursor,
The intention of the operator who is the user is transmitted to the information processing means side by manually operating the mouse or the cursor key.

It is conceivable to use the detection of the operator's line-of-sight direction as means for transmitting intentions which does not depend on such a manual operation. For example, a line-of-sight recognizing device as disclosed in Japanese Patent Laid-Open No. 63-75824 is available. Proposed. The selection of the icon displayed on the display screen and the instruction to move the cursor (pointing) by detecting the line-of-sight direction are
A line-of-sight recognition means for recognizing which position (area) on the display screen the operator is gazing at is required.

When gazing at a certain small area, the eyeball makes an unconscious jumping motion. When stably pointing a small area such as an icon displayed on the display screen using the line of sight, it is necessary to remove such a jump component. However, the known line-of-sight recognition device does not have a mechanism for removing such a jump component.

Further, in order to recognize the position of the line of sight on the display screen, that is, the viewpoint position (hereinafter, the line of sight position and the viewpoint position have the same meaning), on the display screen sent from the eyeball displacement amount detecting means. Numerical information indicating the coordinates of the viewpoint position should be accurately associated with the position of the viewpoint of the operator looking at the display screen. For that purpose, usually, two methods such as the following (1) and (2) are used.

(1) The operator matches the line of sight with the four corners of the display screen in advance, and stores the displacement measurement value, that is, the (coordinate value) sent from the eyeball displacement amount detecting means when the line of sight is adjusted to each point. I'll do it. Then, when recognizing the line of sight of the operator at a certain point in time, the numerical information (coordinate values) at that point and the numerical information at the four corners of the display screen stored in advance are used, and based on the principle of proportional distribution. Recognize the viewpoint position.

(2) The operator focuses his or her eyes on a large number of sample points on the display screen in advance, and stores the numerical information (coordinate values) sent from the eyeball displacement amount detecting device at that time. And when actually using the line of sight,
The sample points on the screen corresponding to the numerical information of the sample points close to the numerical information at that time are recognized as the operator's viewpoint.

In the method according to the above (1), the correspondence relationship between the displacement measurement value sent from the eyeball displacement amount detecting means and the coordinate value defined on the display screen is approximated linearly. However, the geometrical relationship between the eyeball and the display board differs each time the eye camera is worn, and because the surface of the eyeball has a curved surface, the correspondence between the two is generally non-linear. Is low.

On the other hand, in the method (2), the operator must have knowledge in advance at which position on the display screen the information is displayed. However, it is difficult to have such knowledge in advance, for example, when it is necessary to recognize the line of sight of the operator in a situation where the display state on the display screen changes moment by moment.

[0010]

As described above, in the conventional visual axis recognition device, since the output of the eyeball displacement amount detecting means includes the jumping component which is the movement peculiar to the eyeball, the human eye gazes at a certain point of time. It is desirable to remove the noise component such as the jump component in order to correctly recognize the region on the display screen that seems to be moving. Further, from the output of the eyeball displacement amount detection means, which position of the object the operator is looking at, that is, to obtain a non-linear relationship between the eyeball displacement amount and the spatial coordinates defined on the display screen in the conventional gaze recognition device. However, it is difficult to accurately recognize the line of sight of the operator on the display screen.

An object of the present invention is to eliminate the jumping component of the eyeball by temporally smoothing the eyeball displacement detection output from the eyeball displacement detection means, and to make the correspondence between the eyeball displacement and the coordinates on the display screen. Learning using a neural network,
An object of the present invention is to provide a visual axis detection device that can perform stable and accurate visual axis detection by recognizing the visual axis.

[0012]

FIG. 1 is a block diagram showing the configuration of the present invention. In order to achieve the above object, a visual line recognition device (4) of the present invention responds to changes in the visual line of an operator. An eyeball displacement amount detecting means (1) for detecting an eyeball displacement amount, and a displacement amount detection output from the eyeball displacement amount detecting means (1) for smoothing the eyeball displacement amount for a predetermined time. A means (2) and a neural network (neuron circuit network) (3) that receives the eyeball displacement amount smoothing output from the signal smoothing means (2) and learns and recognizes the correspondence with the coordinates on the display screen. It is characterized by

As the eyeball displacement amount detecting means (1), it is preferable to use a known eyeball movement detecting means known as an eye camera, as long as it can detect the displacement of the operator's eyeball. Any means may be used.

[0014]

The eyeball displacement amount detecting means (1) inputs the eyeball displacement amount (V (t) = (Vx (t), Vy (t))) at the time t to the signal smoothing means (2) (however, Vx). (T), V
y (t) corresponds to the lateral and vertical displacements of the eyeball, respectively. The signal smoothing means (2) removes the fine jump component of the operator's eyeball by using not only the actual measurement value but also the past measurement value. By performing such jumping component removal processing, not the eyeball displacement amount at the specific time, but the temporally smoothed average eyeball displacement amount that continues to some extent is detected. This signal smoothing process removes a jump component that occurs when the operator is gazing at a certain point on the display screen.

The neural network (3) learns the correspondence between the eyeball displacement amount smoothing signal output from the signal smoothing means (2) and the viewpoint coordinates of the operator on the display screen.
The X and Y components of the coordinate value on the display screen showing the smoothed eyeball displacement amount are Vx '(t) and Vy' (t). The task that the neural network (3) learns is the coordinates on the display screen from the X and Y components (Vx '(t), Vy' (t)) of the coordinate values on the display screen showing the smoothed eyeball displacement amount. It is a non-linear correspondence to the value (x (t), y (t)).

In general, a neural network can learn an arbitrary non-linear correspondence with high accuracy as the number of intermediate units is increased. Further, from the X and Y components (Vx ′ (t), Vy ′ (t)) of the coordinate value on the display screen showing the smoothed eyeball displacement amount, the coordinate value (x (t), y ( The non-linear correspondence to (t)) is greatly influenced by individual differences of the operator, that is, the eyeball shape (shape of the cornea), the posture with respect to the display screen, and the like.

According to the present invention, the neural network (3) is used to learn this relationship to absorb the individual difference. Further, although the neural network (3) is provided with a plurality of points as points to be learned on the display screen, it is expected that the unlearned points will be output to some extent accurate due to the data complementarity of the neural network. It Therefore, the line-of-sight position in the entire display screen can be accurately recognized by selecting the learning points without deviation from the entire display screen.

The details of the neural network adopted in the present invention are described in, for example, 1989. Publication of industrial books. "Cognitive Science and Search for Neuron Networks", 8th
Chapter "Learning Internal Representation by Error Propagation" (DE Lamelhardt, JL McClelland, RJ Williams, co-authored, translated by Hideki Aso).

[0019]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 2 is a block diagram for explaining the configuration of an embodiment of the line-of-sight recognition device according to the present invention. In this embodiment, the display of the workstation is used as a display screen, and the line-of-sight recognition device is used for icon operation on the screen of this display. The case of using for window operation will be described.

In the figure, 10 is an eye camera as an eye displacement detecting means for detecting the displacement of the operator's eye, 20 is a signal smoothing circuit for smoothing the output signal of the eye camera 1 for a predetermined time, and 30 is Signal smoothing circuit 20
A neural network for learning and recognizing the correspondence between the eyeball displacement smoothing signal output from the operator and the viewpoint coordinates of the operator on the display screen, 40 is a visual line recognition device, 50 is a display control circuit, and 60 is a display.

FIG. 3 is an explanatory view of the screen display of the display 60, 61 is the display screen of the display 60, and 62 is a display screen.
Is an eye cursor controlled by the display control circuit 50, 6
3, 64, 65, 66 and 67 represent various icons.
To move the eye cursor 62 displayed on the display screen 61 of the display 60 to, for example, the icon 63 and select the icon 63, the operator gazes the line of sight toward the icon 63. The direction of the operator's line of sight is detected by the eye camera 10. Eye camera 10
The line-of-sight detected in step 6 is recognized by the signal smoothing circuit 20 and the neural network 30 by the processing described later, and the recognition result is given to the display control circuit 50 to move the eye cursor 62 to the icon 63 and the icon 63. Is selected and processing such as opening the window is performed. Execution of this selection process, for example, the nod of the operator,
Alternatively, the operation is performed by other operation, but this point is outside the scope of the present invention, and thus the description thereof is omitted.

The eye camera 10 is used by being fixed to the operator's head, and includes a light source for projecting light onto the operator's eyeball surface, an image pickup element for receiving reflected light from the eyeball surface, and an image pickup element. And a reflected light position detection circuit for detecting movement of the eyeball based on the image pickup signal. Displacement measurement value V which is the output of this reflected light detection circuit
(T) is the coordinate value (Vx (t), Vy (t)) of the eyeball on the virtual two-dimensional plane as the amount of eyeball displacement.

The signal smoothing circuit 20 has a displacement measurement value V corresponding to the amount of displacement of the operator's eyeball in the horizontal direction (X) and the vertical direction (Y) from the eye camera 10 at a certain time t.
(T) (Vx (t), Vy (t)) is received. In the present embodiment, in smoothing the time of this V (t) time series,
The exponential smoothing method is used. The smoothed value V '(T) of the eyeball displacement at time T is defined by the following recurrence formula.

V ′ (T) = αV (T−1) + βV ′ (T−1) Equation (1) where (α + β = 1) That is, a smoothed value (estimated value) at a certain time It is determined using the measured value and the smoothed value one time before. This method is a smoothing method in which the latest measured value is given more specific gravity.

Then, this V '(T) is measured value V (T)
When expanded only by: V ′ (T) = α [Σ _{k = 0} (1-α) ^k V (T−k)] ... Equation (2) That is, it can be seen that the smoothed value V ^* (T) is the sum of exponentially weighted past measured values. FIG. 4 is a block diagram for explaining the configuration of the smoothing processing circuit 20, in which 21 is a coefficient α storage register for storing the coefficient α, and 2
Reference numeral 2 is a coefficient β storage register that stores the coefficient β, 23 is a smoothed value register, 24 and 25 are multipliers, and 26 is an adder.

FIG. 5 is a flow chart for explaining the operation of the smoothing processing circuit 20. The operation of the smoothing processing circuit shown in FIG. 4 will be described below with reference to the flow chart shown in FIG. As is clear from the circuit shown in FIG. 4, the actual processing of the relationship of the above equation (1) is V (T- of the eye camera 10).
1) is multiplied by the coefficient α from the coefficient α storage register 21 in the multiplier 24, while V ′ from the smoothed value register 23 is multiplied.
This can be realized by a configuration in which (T-1) is multiplied by the coefficient β from the coefficient β storage register 22 by the multiplier 25, and the output values of the multipliers 24 and 25 are added by the adder 26.

That is, in FIG. 5, time (t-1)
At the step S1, the output V (t-1) is received from the eye camera 10 (step S-1), and the output V (t-1) has a coefficient α.
The coefficient α stored in the storage register 21 is multiplied (step S-2). Next, the predicted value V '(t- stored in the smoothed value register 23 at time (t-1)
The coefficient β stored in the coefficient β storage register 22 in 1)
Is multiplied by and added to the result of the above (step S-2),
This is the predicted value V '(t) at time t (step S-3).

Predicted value V'of the result of (step S-2)
(T) is sent to the neural network and the value is stored in the smoothed value register 23 (step S-4).
This process is incremented at time t (step S-5), and the process of step S-1 → step S-2 → step S-3 → step S-4 is repeated to obtain a predetermined time (normally required). The average line-of-sight position of the operator's icon recognition time) is calculated.

The coefficient α defines the smoothness of smoothing. If the α value is too large, the smooth value faithfully reflects the eye movement, but the jump component cannot be absorbed. On the other hand, if the α value is too small, the jump component is averaged and becomes very smooth, but a large time delay occurs with respect to the actually measured value.

Since the appropriate value of the coefficient α depends on the sampling interval of the actually measured value and the characteristics of the eyeball, it is desirable to finely adjust it in actual use. This adjustment can be easily performed by rewriting the coefficient α saving register 21 and the coefficient β saving register 22 in the figure by trial.
Next, the neural network used in this embodiment will be described.

FIG. 6 illustrates a neural network for performing learning and recognition processing of the correspondence between the displacement measurement value (coordinate value) sent from the smoothing circuit 20 and the position of the viewpoint of the operator who is looking at the display screen. In the figure, when the operator sees a certain point (screen coordinate values x, y) on the display screen, the eye camera 10 causes the measurement coordinates, which are displacement measurement values corresponding to the horizontal and vertical displacement amounts of the eyeball. The value (Vx, Vy) is output.

The measured coordinate values (Vx, Vy) and the screen coordinate values (x, y) have a non-linear relationship and are greatly affected by personal factors such as the operator's eye shape and posture. In the present embodiment, a large number of measured coordinate values (Vx, Vy) and screen coordinate values (x, y) are given to the neural network, and learning is performed to acquire a nonlinear correspondence relationship.
FIG. 7 is an explanatory diagram of a sample of learning points on the display screen. 71 to 79 are points to be learned (learning points), 80 is a mark cursor for pointing the learning points, and learning points 71 to 79.
The coordinate value (x, y) on the display screen of is 0 ≦ x ≦
1280, 0 ≦ y ≦ 1280.

Samples of these learning points (sample points)
Are automatically pointed to by the mark cursor 80 shown in the figure from the main body of the workstation, and the operator sequentially gazes at the sample points in accordance with this instruction, so that the coordinates between the measured coordinate values of the eye camera and the coordinates on the screen are displayed. Learn non-linear correspondence. After the neural network executes the above learning, when the operator gazes at an arbitrary position on the display screen, the output (V
From (x, Vy), the neural network can immediately calculate the position (x, y) of the operator's viewpoint.

Processing operation of the neural network of FIG.
Is the i-th sample point (x_i, Y _i) And the strangeness at that time
Position measurement value (Vx_i, Vy_i) When learning the relationship
explain about. The input of the neural network is V
x_i, Vy_i, The output is x '_i, Y '_iBecause it is
Each of the force layer 81 and the output layer 83 is composed of two units.
Is made. The number of the intermediate layers 82 is not limited to the three shown in the figure, and
For more detailed learning by increasing the number of middle layers 82
It can be carried out.

Vx input to the input layer 81_i, Vy
_iIs subjected to nonlinear conversion processing in the intermediate layer 82 and the output layer 83,
X'as the output of the output layer 83_i, Y '_iIs obtained. D
If the Ural network is not learning,
Generally (x '_i, Y '_i ) And (x_i, Y_i) Does not match
Yes. Then, (x '_i, Y '_i) And (x_i, Y _i) And
Neural network is the one described above to match
Learn by law. This learning method is backpropagation
Solution method is used.

When this learning is performed, an evaluation function such as a squared error is set, and the weights between the units are changed so that the value of the evaluation function decreases. In actual learning,
The n pattern pairs are presented to the network with equal probability, and the learning is repeated for each pattern. FIG. 8 is a flow chart for explaining the processing of the neural network.

First, the measured eyeball displacement amount (Vx _i , V
y _i ) is normalized to input layer 8 of the neural network
1 (step S-10). Next, the output values (x ′ _i , y ′ _i ) of the neural network and the coordinate values (x _i , y _i ) of the normalized display screen are compared to reduce the error between the intermediate layer units 82. The weight is changed (step S-11).

Then, the value of i is changed (step S-
12) The process of step S-10 → step S-12 → step S-12 is repeated, and the learning is ended when the errors with respect to all the patterns are below a certain reference value. The operation and learning algorithm of this neural network is outside the scope of the present invention, and the above-mentioned D. E. Since it is described in detail in Ramelhart et al., Its explanation is omitted.

As described above, according to this embodiment, the line-of-sight position in the entire display screen can be recognized accurately.

[0040]

As described above, according to the present invention,
Removing the jumping component of the eyeball by temporally smoothing the eyeball displacement detection output from the eyeball displacement detection means, learning the correspondence between the eyeball displacement and the coordinates on the display screen using a neural network, It is possible to provide a visual line recognition device capable of performing stable and accurate visual line detection by recognizing the visual line.

That is, by adopting a configuration in which the above-mentioned relationship is learned by using a neural network, individual differences of operators can be absorbed, and unlearned points other than the set learning points can be absorbed by the data complementarity of the neural network. It is possible to obtain a recognition output that is accurate to some extent. With this, it is possible to accurately recognize the line-of-sight position in the entire display screen by selecting the learning points without deviation from the entire display screen.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of the present invention.

FIG. 2 is a block diagram illustrating a configuration of an embodiment of a visual line recognition device according to the present invention.

FIG. 3 is an explanatory diagram of screen display on a display.

FIG. 4 is a block diagram illustrating a configuration of a smoothing processing circuit.

FIG. 5 is a flowchart for explaining the operation of the smoothing processing circuit.

FIG. 6 is an explanatory diagram of a neural network that performs learning and recognition processing of the correspondence between the displacement measurement value (coordinate value) sent from the smoothing processing circuit and the position of the viewpoint of the operator who is looking at the display screen. is there.

FIG. 7 is an explanatory diagram of a sample of learning points on the display screen.

FIG. 8 is a flow chart illustrating processing of a neural network.

[Explanation of symbols]

1 ... Eyeball displacement amount detecting means, 2 ... Signal smoothing means, 3 ... Neural network, 4 ...

Claims (1)

[Claims] 1. An eyeball displacement amount detecting means for detecting a displacement amount of an eyeball in accordance with a change of a line of sight of an operator, and a displacement amount detection output from the eyeball displacement amount detecting means for receiving the eyeball displacement amount for a predetermined time. Gaze recognizing device comprising a signal smoothing means for smoothing the eyeballs and a neural network for learning and recognizing the correspondence with the coordinates on the display screen by receiving the eyeball displacement amount smoothing output from the signal smoothing means. ..