SE529156C2 - Computer apparatus controlling system has data-manipulation window presented at position relative to active control object such that center of window is located within small offset distance from center of control object - Google Patents

Abstract

A processing module investigates whether movement sequence of a point of regard (130) fulfills an activation criterion with respect to zone for commands. A data-manipulation window (220a) for visually confirming entered command is presented at a position relative to active control object (220A) such that center of window is located within small offset distance from center of the control object, if activation criterion is fulfilled. A graphics symbolizing activity portion (210) of display in window is updated in response to eye-tracker control entry of control commands. Independent claims are included for the following: (1) method of controlling computer apparatus; (2) computer apparatus controlling program; and (3) computer readable medium storing computer apparatus controlling program.

Description

20 25 30 35 529 156 2 organ. This can lead to fatigue symptoms and a discomfort experienced by the user. Another problem is that the user, who may be a disabled person, may have trouble controlling his gaze with sufficient accuracy to control the computer in the desired way. Of course, this problem is especially emphasized if the eye-steerable screen objects are small. In addition, imperfections in the eye tracking equipment may introduce measurement errors regarding an estimated point of view on the screen. In order to improve the human-machine interface, U.S. Patent No. 6,152,563 proposes that the so-called collapsing symbols are presented over selected window objects in order to visually confirm the selection of the object for a period before any manipulation. This allows the user to cancel incorrect selections before performing the manipulation with respect to the object. The document also suggests the use of zoom windows to improve the precision of the user's gaze fixations on certain screen objects.

However, it is not only difficult to estimate the user's point of view adequately. Another dilemma in eye tracking is that it can be problematic for the user to select a screen object. Expressing the user's intentions for the computer via an eye-guided interface can be difficult for many reasons. An important aspect is the fact that the eyes are designed for perception, not for motor tasks, and that the gaze therefore moves across the screen even when the user is not interested in generating any control commands. A dwell time, that is, the time during which the user fixes his gaze on an object on the screen, can be used to express the user's intention to manipulate certain objects. A disadvantage of this approach is that the interaction process becomes relatively slow, and risks irritating the user. Confirmation commands in the form of flashes can speed up the process significantly. However, human winks are often unintentional, so the user could generate numerous unwanted control commands. Of course, a physical key, or key, offers a much more reliable means of confirmation. Nevertheless, this may not be open to the user, who may be incapable of manipulating such buttons / keys. Acknowledgment via voice recognition interface is another option, which is less reliable, and may not be a reasonable choice for the user due to physical disabilities, or factors in the environment in which the system is used, such as noise.

Human gaze patterns include so-called saccades, which represent rapid movements of the black from one point to another. These movements are almost never involuntary, except in nystagmus patients. However, using sacks to enter data or commands via an eye-steerable input interface is problematic for other reasons. Namely, if a sack activates a function with respect to a certain screen object, the sack itself typically results in the user no longer observing this object, at least not directly. Consequently, further eye-guided manipulation becomes very difficult, if at all possible. Again, this is a problem related to combined motor and perceptual tasks caused by the eye-controllable input interface.

The cursor symbol is generally an important object to control when interacting with a computer. However, this is not simple either.

The most high-resolution part of the retina, the macula (or macula), has a diameter corresponding to one degree of image angle.

Therefore, a person never needs to direct his gaze with higher precision than that given by this measure, that is, within a degree. Thus, a perfect match between the user's gaze and a cursor symbol on the screen cannot be achieved, no matter how high the accuracy of the eye follower. Instead, a cursor symbol that is controlled directly in response to an estimated point of view must necessarily introduce artifacts, which are perceived as faulty by a human user. Nevertheless, U.S. Patent No. 6,637,883 discloses an eye tracking system for displaying a video screen pointer at a user's point of view. For improved precision, here is a camera which registers the user's eye movements placed in one of the user's purely worn spectacle frames. Of course, this results in special problems, e.g. in terms of fit and comfort. instead of controlling the cursor symbol as a direct response to the estimated point of view, relative cursor control is normally preferred, i.e. solutions where the cursor is caused to move indirectly by manipulating certain controls on the screen for each up / down direction and right left. When placing the gaze point on such a control member, however, the user can no longer see the cursor symbol to be controlled.

U.S. Patent No. 6,204,828 discloses an integrated gaze / manual cursor positioning system which assists an operator in positioning a cursor by integrating a gaze signal and a manual input. When a mechanical activation of an operator device is detected, the cursor is placed at an initial position that is predetermined in relation to the operator's current gaze area. This provides a user-friendly cursor function, but no cursor-based control commands are input to the computer based on the gaze signal.

SUMMARY OF THE INVENTION The object of the present invention is therefore to provide an efficient, reliable and user-friendly eye-tracking solution, which alleviates the above problems and enables an adequate information feedback to the user without sacrificing the user's ability to control the events on the screen.

According to one aspect of the invention, the object is achieved by the system initially described, wherein the processing module is adapted to examine whether the motion sequence of the gaze point meets an activation criterion with respect to an activation zone for any of at least one control command. If the processing module finds that such a criterion is met, the module presents a data manipulation window at a position relative to an active control object on the screen, so that a center point of the window is within a relatively small displacement distance from a center of the active control object. For example, the data manipulation window may be presented in the active control object on the screen. However, a window position immediately adjacent to, or partially overlapping, the active control object is equally conceivable.

The window must at least be positioned so that the user can actively observe both the active control object and the data manipulation object at the same time. The data manipulation object includes graphical information that symbolizes an activity part of the screen which is currently subject to eye-tracking input of control commands. The processing module is also adapted to repeatedly update the information in the data data manipulation window in response to the eye control controlled entry of control commands.

An important advantage of this system is that the user obtains adequate information regarding the manipulation object while concentrating his / her gaze on the control object. This ensures an efficient and comfortable human-machine interaction.

According to a preferred embodiment of this aspect of the invention, the processing module is adapted to determine the active control object based on the movement sequence of the gaze point in relation to the activation zone, for example on the basis of a fixation and a soft tracking track. Preferably, however, a registered first sag of the gaze point from the activation zone indicates the active control object. This identifies the active control object in a very intuitive and user-friendly way.

According to another preferred embodiment of this aspect of the invention, the at least one control command expresses a navigation direction of a cursor symbol with respect to the screen, a rotation of at least one screen object representation of at least one graphical user interface component (GUI component) on the screen, a color change of the screen at least one screen object representation of at least one graphical GUI component on the screen and / or a texture modification of at least one screen object representation of at least one graphical GUI component on the screen. Thus, a great deal of freedom is offered in terms of the manipulation made possible via the proposed interface.

According to yet another preferred embodiment of this aspect of the invention, the control command is assumed to specifically express a navigation direction for a cursor symbol with respect to the screen. Here, the processing module is adapted to distinguish a navigation direction from the first sacked relative to the activation zone and move the cursor symbol according to this navigation direction. This has been shown to offer a highly efficient relative cursor control function.

According to a further preferred embodiment of this aspect of the invention, the processing module is adapted to present a set of control objects on the screen, where each control object in the set is adapted to indicate a respective navigation direction for the cursor symbol in relation to an information area. For example, a first control object may be adapted to indicate a cursor symbol movement to the right, a second control object may be adapted to indicate a cursor symbol movement downwards, and so on. This presents a robust and reliable control interface for the user.

According to a first preferred embodiment of this aspect of the invention, the processing module is adapted to present a set of control objects on the screen in such a way that said set moves gradually across the screen in accordance with the direction of navigation. In addition, the processing module is adapted to move the cursor across the screen in accordance with the direction of navigation. This eye tracking-based cursor symbol control is generally advantageous when the largest possible active screen area is sought.

According to a second preferred embodiment of this aspect of the invention, the processing module is instead adapted to present a set of control objects at fixed positions on the screen. Here too, however, the processing module is adapted to move the cursor symbol across the screen in accordance with the navigation direction. This offers an alternative eye-controllable means for the user to influence the position of the cursor symbol while he / she has the opportunity to visually register this movement, which is desirable in applications where the entire screen area is not requested for utility data.

According to another preferred embodiment of this aspect of the invention, an information area on the screen is assumed to present editable symbols, for example representing text.

The processing module is here adapted to determine at least one added symbol based on a motion sequence of the point of view over a number of screen objects visually representing GU1 components of a software keyboard presented on the screen. The processing module is further adapted to add at least one added symbol to the symbols in the information area. This allows the user to efficiently enter text into the computer.

According to yet another preferred embodiment of this aspect of the invention, the processing module is adapted to examine whether the eye point meets an activation criterion with respect to an active control object representing an erase button. If such a criterion is found to be met, the processing module is adapted to delete a subset of symbols, the subset comprising at least one of the editable symbols. Thus, the user can also conveniently edit an existing text stored in the computer.

According to yet another preferred embodiment of this aspect of the invention, the set of control objects comprises a single control object. Here, the machining module is adapted to determine a target location for a first sacked outside the activation zone. The processing module is adapted to present the single control object at the target location. The single control object again includes the data manipulation window. In analogy to the above, the processing module is adapted to move the cursor symbol across the screen in accordance with a navigation direction expressed by the first sack. This construction is desirable because it allows the user to achieve the desired cursor navigation direction without the user having to move his gaze point after the first sack. In addition, the navigation direction can be chosen freely.

According to a further preferred embodiment of this aspect of the invention, the processing module is adapted to examine whether the eye point meets a deactivation criterion with respect to the control object object. If the processing module finds that such a criterion is met, the module is adapted to deactivate the active control object. Typically, this deactivation criterion is equivalent to a completed manipulation and / or that another manipulation is requested.

According to another aspect of the invention, the object is achieved by the method initially described, in which it is examined whether the movement sequence of the eye point meets an activation criterion with respect to an activation zone for at least one control command. If such an activation criterion is found to be met, the method involves presenting a data manipulation window at a position relative to an active control object on the screen so that a center point of the window is within a relatively small displacement distance from a center point of the active control object. This window in turn includes graphic information which symbolizes an activity part of the screen which is currently subject to eye control input of control commands.

In addition, the method involves repeatedly updating the information in the data manipulation window in response to the eye-controlled input of control commands.

The advantages of this method as well as the preferred embodiments thereof appear from the discussion above with reference to the proposed system. According to a further aspect of the invention, the object is achieved by a computer program, which is directly loadable to the internal memory of a computer, the program including software for controlling the method proposed above when the program is run on a computer.

According to another aspect of the invention, the object is achieved by a computer-readable medium having a program stored thereon, the program being adapted to cause a computer to control the method proposed above. Additional advantages, advantageous features and applications of the present invention will become apparent from the following description. and the dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will now be explained in more detail by means of preferred embodiments, which are described by way of example, and with reference to the accompanying drawings.

Figure 1 shows a schematic view of an eye tracking system according to an embodiment of the invention; Figures 2a-b show a first use of a first set of control objects according to an embodiment of the invention; Figure 3 shows a second use of the first set of control objects shown in Figure 2a-b; Figure 4 illustrates a use of a second set of control objects according to an embodiment of the invention; Figure 5 illustrates a use of a third type of control object according to an embodiment of the invention; Figure 6 shows a display object implementing a software keyboard, which is controllable in response to a sequence of movements of a user's point of view according to an embodiment of the invention; Figure 7 illustrates how a graphic object can be rotated by means of a control object according to an embodiment of the invention; illustrates how the color of a graphic object can be modified by means of control objects according to an embodiment of the invention; and Figure 8 Figure 9 illustrates, by means of a flow chart, a general method according to the invention.

DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION We first refer to Figure 1, which shows a schematic view of an eye tracking system according to an embodiment of the invention. The system includes a computer apparatus 140 and an associated screen 150. It is assumed that the computer apparatus 140 presents at least one screen representation of at least one GUI component on the screen 150, the at least one GUI component being controllable in response to commands from an eye. follow-up bodies. The eye tracking means includes an imaging means 160, for example including a video camera. Preferably, the eye tracking means also includes one or more light sources, for example a first set of light sources 170 located relatively close to an optical axis of the imaging means 160, and a second set of light sources 181 and 182 located relatively far from the optical axis of the imaging means 160. This enables both light pupil tracking (BP = Bright Pupil; PCCR tracking) and dark pupil tracking (DP = Dark Pupil; PCCR tracking) of a user's 100 eyes 101 and 102. In any case, the eye tracking means is adapted to repeatedly determine the eye tracking point 130 of the user 100 on the screen 150. The eye tracking means is further adapted to input control commands to the computer apparatus 140 in response to motion sequences of the eye point 130 across the screen 150. To this end, the eye tracking means may include a processing module. 145, which is adapted to receive eye tracking data DV and extract the control commands therefrom. The processing module 145 is further adapted to generate graphical feedback information DFB for presentation on the screen 150 to visually confirm any input control commands.

According to the invention, the processing module 145 is further adapted to cause a data manipulation window to be presented on the screen 150. This window includes graphical information, which symbolizes a part of the screen 150 containing objects based on which control commands are input to the computer 140.

This means that the data manipulation window either includes duplicate information of the graphic data within the activity part 210, an icon which schematically represents this data, or some other kind of unambiguous image of the data. The details of the presentation and control procedure will be discussed in detail below with reference to Figures 2 to 9.

Figure 2a shows a part 200 of an image surface of the screen 150 in figure 1. The screen part 200 contains a set of control objects 220A, 2208, 220C and 220D, which are adapted to present a respective data manipulation window 220a, 220b, 220c and 220d.

Preferably, each object in the set is adapted to indicate a respective navigation direction for a cursor symbol in relation to an information area on the screen 150, for example to the right, downwards, to the left and upwards, respectively. Preferably, however, no more than one control object at a time presents such a data manipulation window, and before any eye control-based entry of control commands has begun, no data manipulation window is displayed at all. In all cases, the set of control objects 220A, 220B, 220D and 220D is centered around a point 221 in a so-called activity part 10 15 20 25 30 529 156 12 210. The activity part 210 is an area of the screen 150 which contains one or more graphic elements with with respect to which eye-tracking generated command commands are generated and input to the computer apparatus 140. The activity part 210 is thus centered around a screen object with respect to which control commands can be input to the computer apparatus 140.

Referring to Figure 2, we will now explain how the activity portion 210 is selected according to a preferred embodiment of the invention for controlling a cursor symbol 231 over the image area of the screen. Figure 2b shows a set of control objects 220A, 220B, 220C and 220D superimposed on an information area 240 containing editable text. The processing module 145 is adapted to examine whether the gaze point 130 meets an activity criterion. This criterion here refers to the cursor symbol 231 and its desired movement. The activity criterion is considered specifically met if the gaze point 130 has been within an activity zone 210 around the cursor symbol 231 during at least one activation interval, i.e. the user's gaze has been directed sufficiently close to the cursor symbol for a sufficiently long time. If the processing module 145 finds that this criterion is met, the module 145 presents the set of control objects 220A, 220B, 220C and 220D as shown in Figure 2b, i.e. centered around the cursor symbol 231. Thus, the activity part 210 is represented here by an area on the left. and to the right of the cursor symbol 231, which is located between the control objects 220A, 220B, 220C and 220D. Since the cursor symbol 231 in this case is a text entry cursor, it is also possible to manipulate the text to the left and to the right of the cursor symbol 231, i.e. delete symbols and / or add new symbols.

Of course, the invention does not preclude that the processing module 145 causes one or more of the control objects 220A, 220B, 220C and 220D to be presented also under other conditions, for example constant, regardless of whether the activation criterion is met or not. 10 15 20 25 30 35 529 156 13 nevertheless, we assume here that the user 100 after the above-mentioned activation moves the gaze point 130 by means of a sack in a general right-hand direction. According to this embodiment of the invention, the processing module 145 is adapted to interpret such a slow motion that the user wishes to move the cursor symbol 231 to the right across the screen. Thus, the processing module 145 detects the sacked and interprets the sacking as an expression of a navigation direction RW to the right of the screen 150. As a result, the processing module 145 presents a data manipulation window 220a in the control object 220A on the screen 150, which is adapted to move to the right The window 220a includes graphical information that duplicates the activity part 210, which in this example means that the window 220a shows the letters "nde", i.e. two letters "nd" to the left of the cursor symbol 231 and a letter "e" to the right of the cursor symbol. 231. Of course, any larger or slightly smaller window 220a is equally conceivable, as well as various kinds of gradually faded or partially hidden representations of the letters in the window 220a in the vicinity of the right and left window edges.

According to the invention, the processing module 145 is adapted to repeatedly update the information in the data manipulation window 220a in response to eye-tracking input of control commands to the computer 140. This means that the two letters to the left and the single letter to the right of the cursor symbol 231 are changed accordingly. the content of the text in the information area 240 as the cursor symbol 231 moves across the screen 150 in accordance with what the navigation direction RW indicates.

According to a preferred embodiment of the invention, the processing module 145 is also adapted to present the set of control objects 220A, 220B, 220C and 220D on the screen 150 in such a way that this set moves gradually over the screen 150 in accordance with the navigation direction RW. This means that the relative positional relationship between the cursor symbol 231 and the set of control objects 220A, 220B, 220C and 220D remains constant, but that they are all moved together across the screen 150. This provides a stable control mechanism. of the cursor symbol, which is desirable even for non-text applications.

Figure 3 illustrates such an example, in which the set of control objects 220A, 220B, 220C and 220D shown in Figures 2a and 2b is used to control a cursor symbol 331 with respect to a portion 200 of an image area of the screen 150 containing an information area. in the form of a window 340.

In analogy to the above, we assume that the user 100 has placed the gaze point 130 within an activation zone around the cursor symbol 331 during at least one activation interval. Therefore, the processing module 145 has caused the presentation of the set of control objects 220A, 220B, 220C and 220D around an activity area 210 containing the cursor symbol 331.

In addition, the user has performed a first swipe to the right to express a navigation direction RW, and as a result, the processing module 145 has activated a control object 220A adapted to effect a rightward movement of the cursor symbol 331 and presented therein a data manipulation window 220a. Furthermore, we assume that the processing module 145 has examined whether the user 100's focus point 130 has been on the active control object 220A during a threshold time, and found this criterion to be met. Therefore, the processing module 145 has initiated a cursor movement to the right over the screen 150, i.e. in accordance with the navigation direction RW. Again, the presented data manipulation window 220a includes graphical information symbolizing the activity portion 210. In contrast to the example discussed above with reference to Figure 2b, the cursor symbol 331 is controlled in response to eye-tracking generated control commands which are adapted to manipulate graphical objects, such as windows. , software-implemented buttons, scroll bars, sliders, etc. Nevertheless, the cursor symbol 331 remains within the activity part 210 because the cursor symbol 331 is an object based on which control commands are entered into the computer 140. The processing module 145 also updates repeatedly the information in the data manipulation window 220a in response to generated control commands (here represented by the navigation direction RW), so that when the cursor symbol 331 moves to the right over the image surface of the screen 150 the contents of the window 340 existing to the right of an original position gradually appear in the data manipulation window 220 a.

Figure 4 illustrates how an alternative set of control objects 420A, 420B, 42OC and 420D can be used according to an embodiment of the invention to control the cursor symbol 431 over the screen 150. Here the screen 150 shows an information area containing a window 440. However, this embodiment is equally applicable to other types of information, such as the text 240 in Figure 2b.

We again assume that the user 100 has placed the gaze point 130 within an activation zone 430 around the cursor symbol 431 during at least one activation interval. Therefore, the processing module 145 has caused the set of control objects 420A, 420B, 42OC and 420D to be presented on the screen 150, so that a certain control object is located near the respective side of the screen 150. Alternatively, the control objects 420A, 420B, 42OC and 420D are always presented on inserts 150 regardless of whether or not the above activation criterion is met. In any case, a first control object 420A is located at a fixed position on the right side of the display 150 and is preferably adapted to effect a movement to the right of the cursor symbol 441, a second control object 420B is located at a fixed position. position on the lower side of the screen 150 and is preferably adapted to effect a downward movement of the cursor symbol 441, a third control object 42OC is located at a fixed position on the left side of the screen 150 and is preferably adapted to causing a leftward movement of the cursor symbol 441, and a fourth control object 420D is at a fixed position on the upper side of the screen 150 and is preferably adapted to effect an upward movement of the cursor symbol 441.

In addition, the user has performed a first downward descent to express a desired downward DW navigation direction. As a result, the processing module 145 has activated the second control object 420B and presented therein a data manipulation window 420b. Furthermore, we assume that the user 100 has fixed his gaze point 130 on the active control object 420B during a threshold time, and that therefore the processing module 145 has initiated a downward movement of the cursor symbol over the screen 150.

Accordingly, the data manipulation window 420b includes graphical information that initially duplicates a screen area, for example, substantially corresponding to the activation zone 430. reflect the downward movement of the cursor symbol 431 in the navigation direction DW, preferably while holding a representation of the cursor symbol 431 at a constant position in the window 420b.

According to a preferred embodiment of the invention, the processing module 145 is adapted to examine whether the focal point 130 has been outside the active control object 420B during a motion deactivation period, say in the order of 0.5 seconds. If this criterion is met, the processing module 145 is adapted to deactivate the active control object 420B and thus interrupt the eye-tracking controlled entry of control commands, i.e. in this example the downward movement of the cursor symbol 431.

This stop function, which is applicable to all embodiments of the present invention, offers a very effective way for the user 100 to indicate to the system that the current input of control commands should stop, and that possibly another kind of command instead to be generated.

Figure 5 illustrates how a third type of proposed control object 520A may be used in accordance with an embodiment of the invention to affect a cursor symbol 531 based on eye tracking generated command commands input to the computer 140. According to this embodiment, the set of control objects includes a single control object 520A, can be presented in any place of the screen 150 image area.

After the processing module 145 has determined that the user 100 has placed his focal point 130 within the activation zone 530 around the cursor symbol 531 for at least one activation interval, the processing module 145 is specifically adapted to not only determine a direction 130d for a first sagged 130d outside the activation zone. 530, but also a target / aceration of such a sack. The processing module 145 is then adapted to present the control object 520A at this position, for example centered above it. In analogy to the embodiments described above, the control object 520A includes a data manipulation window 520a, which includes graphical information symbolizing an activity portion of the screen 150 which is currently subject to eye-tracking input of control commands to the computer 140, namely in this example, cursor symbol 531. In addition, the processing module 145 is also adapted here to move the cursor symbol 531 over the screen 150 in accordance with the navigation direction 130d. It is worth mentioning that the navigation direction can have any angle (0 ° -360 °) in relation to a reference direction, and is thus not limited to the previously described horizontal and vertical right / left and down / up directions. - the ings. Figure 6 shows a display object that implements a software keyboard 650 on the screen 150, the keyboard 650 according to an embodiment of the invention being controllable in response to a motion travel sequence of the user 100's focus point 130. The keyboard 650 is associated with an information area 640, which is adapted to present editable symbols, for example, representative text. A cursor symbol 631 indicates a position in the information area 640 that is currently manipulable.

The keyboard 650 includes a set of keys 620 adapted for eye control to enter alphabetic symbols, numbers and other types of characters into the computer 140. The keyboard 650 may also include a number of display objects 620A, 621 which 622 visually represent GUI components. of software keys on the screen 150, which enable the user 100 to perform various kinds of editing functions based on eye-tracking generated control commands. Thus, objects 620, 620A, 621 and 622 may be control objects of the type described above.

The processing module 145 is here adapted to determine whether the user 100 has placed his eye point 130 in relation to these control objects, so that an activation criterion with respect to a certain control object has been fulfilled. Provided that a specific control object is activated, the resulting manipulation of the cursor symbol 631. The cursor symbol 631 can in turn either be moved relative to the information area 640 as described above, for example with reference to Figures 2a and 2b, or by means of arbitrary other known control means.

Assuming the user has steered the cursor symbol 631 to a desired position in the information area 640, the processing module 145, in analogy to the above, then examines whether an activation criterion with respect to any of the control objects 620, 620A, 621 and 622 is met. For purposes of illustration, we assume in this case that a backspace / delete key 620A has been activated in this way based on a travel sequence of an eye point 10 15 20 25 30 529 156 19 130 relative to an activation zone 630 for this key 620A, e.g. the user 100 has placed the gaze point 130 within the activation zone 630 for at least one activation interval. We also assume that the cursor symbol 631 is located between the letters “ö” and “r” in the word “for” as shown in Figure 6, and that thus an activity part 210 of the information area 640 is defined. Since the key 640A is adapted to effect an erasure of the symbol located immediately to the left of the current position of the cursor symbol 631, this symbol, i.e. here the letter “ö”, will be erased in connection with the activation of the key 640A. As a further result, the cursor symbol 631 will then be located between the letters "f" and, like the embodiment of the invention described above with reference to Figure 2b, includes a data manipulation window 620a in the active control object initially represented by the key 640A. when activating graphic information symbolizing an activity part around the cursor symbol 641. In this example, the window 620a shows the letters "for", ie two letters "fö" to the left of the cursor symbol and a single letter "r" to the right of the cursor symbol 631. Again any smaller or larger window 620a adapted to the key 620A as conceivable.In addition, the symbols in the window 620a may be displayed so that they gradually fade out, or are partially hidden near one, or both, of the right and left edges of the window 620a.

Of course, when selecting one of the other keys 620, the processing module 145 may instead determine that one or more symbols should be added based on the motion sequence of the gaze point 130 across the keyboard 650, and add such added symbols to the information area 640. In this embodiment of the invention, a number, preferably all, of the keys 620, 620A, 621 and 622 are adapted to present a respective data manipulation window in case the key is selected as the active control object. Figure 7 illustrates how a screen representation 740 of a GUI component can be manipulated according to an embodiment of the invention, namely subjected to a rotation. According to this embodiment, a first control object 720A and a second control object 72OB are presented on the screen. Upon activation, the first control object 720A is adapted to effect a clockwise rotation of the screen representation of a selected graphic object, while the second control object 72OB is adapted to effect a counterclockwise rotation. Here, one control object is activated, say the second control object 72OB by moving the gaze point 130 across the screen so that the gaze point 130 meets an activation criterion with respect to a graphic representation 740, for example by the gaze point 130 being within an activation zone 730 during an activation interval. Then, the gaze point 130, typically in the form of a sack, moves to the second control object 72OB. Provided that the focal point 130 is on this object 72OB at least during a threshold time, a data manipulation window 720b is presented in the second control object 72OB to confirm activation. The data manipulation window 720b displays graphical information that symbolizes the graphical representation 740, that is, the current object of eye tracking controlled counterclockwise rotation.

In addition, as the graphical representation 740 is rotated on the screen, the information reflecting this fact in the data manipulation window 720b is repeatedly updated. Preferably, the rotation continues gradually as long as the control object 72OB remains active, either in relatively small steps (eg 1 degree per update at an update rate of 4 updates per second), or in relatively large steps (eg 90 degrees per update at an update rate of one per second). Whenever the user is satisfied with the rotation, he / she moves the gaze point 130 so that a deactivation criterion is met, for example by placing the gaze point 130 outside the control object 72OB during a deactivation threshold time. Figure 15 illustrates yet another type of eye control manipulation of graphic objects made possible in accordance with an embodiment of the present invention. Here, a screen representation 840 of a GU1 component can be controlled to change color. A first control object 820A is adapted to color a selected object red, a second control object 8208 is adapted to color a selected object blue, and a third control object 82OC is adapted to color a selected object green. Again, a given graphical representation 840 is selected by moving the gaze point 130 across the screen so that the gaze point 130 meets an activation criterion with respect to the given graphic representation 840. Preferably, the activation criterion is considered met if the gaze point 130 has been located within an activation zone 830 around it. given graphic object 840 during an activation interval.

The eye point 130 is then moved by means of a sack to a control object, say the second control object 820B, where the eye point 130 remains for at least one threshold time. Thereafter, the modification of the graphical representation 840 indicated by the second control object 820B, in this case a color change to blue, is effected. To confirm this modification, a data manipulation window 820b is presented which displays graphical information symbolizing the graphical representation 840 in the second control object 820B. Thus, in connection with the graphical representation 840 changing color to blue, this fact is reflected by symbolic information in the data manipulation window 820b.

Of course, analogous to the above, other properties of a GUI component can also be changed, such as textures. Therefore, according to an embodiment of the invention, a set of eye-controllable control objects is presented on the screen, which are adapted to influence that a certain texture is imaged on the screen representation of the GUI component.

To sum up, the general method according to the invention for controlling a computer apparatus will now be described with reference to the flow chart in Figure 9. An initial step 910 determines a user's point of view on the screen. A step 920 then examines whether an activation criterion has been met, and if so, a step 930 follows. can be input to a computer.

Alternatively, the activation criterion can be considered fulfilled if the focal point during the activation interval has been on the control object more than on any of its adjacent control objects. However, if the activation criterion is not met, the procedure loops back to step 910.

Step 930 defines the control object as active and presents a data manipulation window relative to the active control object on the screen so that a center point of the window is within a relatively small displacement distance from a center point of the active control object. For example, the data manipulation window may be presented in the active control object, or so that the window partially overlaps the active control object. The control object to be activated can be identified by a first sacked at the point of view registered from the activation zone, and the control object representing the at least one control object the command command to be input to the computer. The data manipulation window includes graphical information, which symbolizes an activity part of the screen where the screen object is located on which eye control-controlled commands are currently generated.

Next, a step 940 examines whether a deactivation criterion is met with respect to the active control object. Preferably, the deactivation criterion is the complement to the above-mentioned activation criterion. Thus, the deactivation criterion is considered to be met if the activation criterion is no longer met. Of course, a deactivation criterion for a first active control object is generally considered to be met if an activation criterion for a second active control object is met. In all cases, the procedure loops back to step 910 if the deactivation criterion is met. In another case, a step 950 follows in which the information in the data manipulation window is updated in response to the at least one control command represented by the active control object.

All the method steps, as well as any sub-sequence of steps, described with reference to Figure 9 above can be controlled by means of a programmed computer apparatus. In addition, although the embodiments of the invention described above with reference to the figures include a computer and processes performed in a computer, the invention extends to computer programs, especially computer programs on or in a carrier adapted to practically implement the invention. The program may be in the form of source code, object code, a code which constitutes an intermediate between source and object code, such as in partially compiled form, or in any other form suitable for use in implementing the process according to the invention.

The carrier can be any entity or device which is capable of carrying the program. For example, the carrier may comprise a storage medium such as a flash memory, a ROM (Read Only Memory), for example a CD (Compact Disc) or a semiconductor ROM, an EPROM (Erasable Programmable ROM), an EEPROM (Electrically EPROM) or a magnetic recording medium, for example, a floppy disk or hard disk. In addition, the carrier may be a transmitting carrier such as an electrical or optical signal, which may be conducted through an electrical or optical cable or via radio or otherwise. When the program is formed by a signal which can be conducted directly by a cable or other device or means, the carrier can be constituted by such a cable, device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, where the integrated circuit is adapted to perform, or to be used in performing, the actual processes.

The term "includes" when used in this specification is understood to indicate the presence of the said features, integers, steps or components. However, the term does not exclude the presence or addition of one or more additional features, integers, steps or components. 529 156 24 The references to prior art in this specification are not, and should not be construed as, an acknowledgment or indication that said technology is part of the general knowledge of Australia.

The invention is not limited to the embodiments described in the figures, but can be varied freely within the scope of the claims.

Claims (26)

10 15 20 25 30 35 529 156 25 Patent claims A system for controlling a computer apparatus (140) connected to a graphic screen (150), the computer apparatus (140) being adapted to present a respective screen representation of at least one GUI component on the screen (150), the system comprising eye tracking means ( 145, 160, 170, 181, 182) adapted to determine a user's (100) point of view (130) on the screen (150) and in response to motion sequences of the point of view (130) across the screen (150) input control commands to the computer (140) , wherein the eye tracking means comprises a processing module (145) adapted to generate graphical feedback information (DFB) for presentation on the screen (150) to visually confirm input commands, characterized in that the processing module (145) is adapted to: examine whether the motion sequence of the flash point (130) satisfies an activation criterion with respect to an activation zone (430, 530, 630, 730, 830) for each of the at least one control commands, and if such activation s criterion is found to be satisfied presenting a data manipulation window (220a, 420b, 520a, 620a, 720b, 820b) at a position relative to an active control object (220A; 420B; 520A; 620A, 720B, 820B) on the screen (150) so that a center point of the window is within a relatively small displacement distance from a center point of the active control object, the window including graphical information symbolizing an activity portion (210) of the screen (150). ) which is currently subject to eye control input of control commands, and repeatedly updating the information in the data manipulation window (220a, 420b, 520a, 620a, 720b, 820b) in response to the eye-guided input of control commands. The system of claim 1, characterized in that the processing module (145) is adapted to determine the active control object (220A; 420B; 520A; 620A, 720B, 820B) based on the motion sequence of the gaze point (130) relative to the activation zone (430). , 10 15 20 25 30 529 156 26 530, 630, 730, 830). The system of claim 2, characterized in that the motion sequence of the gaze point (130) based on which the machining module (145) is adapted to determine the active control object (220A; 420B; 520A; 720B, 820B) comprises a first sacked from activation zones (430, 530, 730, 830). The system according to any of the preceding claims, characterized in that it at least one control command expresses at least one of: a navigation direction (RW, DW, LW, 130d) of a marker symbol (231, 331, 431, 531, 631) with respect to on the screen (150), a rotation of at least one screen object representation (740) of at least one GUI component on the screen (150), a color change of at least one screen representation (840) of at least one GUI component on the screen (150), and a texture modification of at least one screen representation of at least one GUI component on the screen (150). The system according to claim 4, characterized in that at least one control command expresses a navigation direction (RW, DW, LW, 130d) of a cursor symbol (231, 331, 431, 531, 631) with respect to the screen (150) and the processing module (145) is adapted to: distinguish a navigation direction (RW, DW, LW, 130d) from a first sag relative to the activation zone (430, 530), and move a cursor symbol (231, 331, 631) across the screen (150) accordingly with the navigation direction (RW, DW, LW, 130d). The system of claim 5, characterized in that the processing module (145) is adapted to present a set of control objects (220A, 220B, 220C, 220D; 420A, 420B, 42OC, 420D; 520A; 620) on the screen (150), where each control object in the set is adapted to indicate a respective navigation direction 10 15 20 25 30 529 156 27 (RW, DW, LW, 130d) for a cursor symbol symbol (231, 331, 431, 531, 631) relative to an information area ( 240, 340, 440, 540, 640) The system according to claim 6, characterized in that the processing module (145) is adapted to: present the set of control objects (220A. 220B. 2200. 220D) on the screen (150) in such a way that said set is gradually moved across the screen (150) according to the navigation direction (RW), and move the cursor (231, 331, 431, 531, 631) across the screen (150) according to the navigation direction (RW). The system according to claim 6, characterized in that the processing module (145) is adapted to present the set of control objects (420A, 420B, 420C, 420D; 620) at fixed positions on the screen (150), and to move the cursor (431, 631) across the screen (150) according to the navigation direction (DW, LW). The system according to any of the preceding claims, characterized in that the screen (150) presents an information area (640) comprising editable symbols, and that the processing module (145) is adapted to: determine at least one added symbol based on a sequence of motion of the gaze (130) over a number of screen objects (620, 621, 622) visually representing GUI components of an on-screen software keyboard (150), and adding at least one added symbol to the symbols in the information area (640). The system according to any of the preceding claims, characterized in that the screen (150) presents an information area (640) comprising editable symbols, and that the processing module (145) is adapted to: examine whether the gaze point (130) meets an activation criterion with respect to an active control object (620A) representing a delete button, and if such a criterion is found to be deleted, delete a subset of symbols, the subset comprising at least one of the editable symbols. The system according to any one of claims 1 to 8, characterized in that the set of control objects comprises a single control object (520A), and the processing module (145) is adapted to: determine a target location for a first sack (130d) outside the activation zone (530), present the single control object (520A) at the target location, the single control object (520A) including a data manipulation window (520a), and moving the cursor symbol (531) across the screen (150) according to a navigation direction (130d) expressed by the first sack. The system according to any one of the preceding claims, characterized in that the processing module (145) is adapted to: examine whether the gaze point (130) meets a deactivation criterion with respect to the control object (220A; 420B; 52OA; 620A, 720B, 820B), and if such a criterion is found to be met, deactivate the active control object (220A; 420B; 520A; 620A, 720B, 820B) A method of controlling a computer apparatus attached to a graphical display (150), the computer apparatus being adapted to present a respective screen representation of at least one GUI component on the screen (150), the method comprising: determining a user's ( 100) focus point (130) on the screen (150) and in response to motion sequences of the point of view (130) across the screen (150) input control commands to the computer (140), generating graphical feedback information (DFB) for pre- Position on the screen (150) to visually confirm input commands, characterized by examining whether the motion sequence of the gaze (130) meets an activation criterion with respect to an activation zone (430, 530, 630, 730, 830) for each and one of the at least one control commands, and if such an activation criterion is found to be met, presenting a data manipulation window (220a, 420b, 520a, 620a, 720b, 820b) at a position relative to an ak active control object (220A; 420B; 520A; 620A, 720B, 820B) on the screen (150) so that a center point of the window is within a relatively small displacement distance from a center point of the active control object, the window including graphical information symbolizing an activity portion (210) of the screen (150) which is currently subject to eye control input of control commands, and repeatedly updating the information in the data manipulation window (220a, 420b, 520a, 620a, 720b, 820b) in response to the eye-guided input of control commands. The method according to claim 13, characterized by determining the active control object (220A; 420B; 520A; 620A, 720B, 820B) based on the motion sequence of the gaze point (130) relative to the activation zone (430, 530, 630, 730, 830). The method of claim 14, characterized in that the motion sequence of the gaze point (130) based on which the processing module (145) is adapted to determine the active control object (220A; 420B; 520A; 720B, 820B) comprises a first sacked from activation zones (430, 530, 730, 830). The method according to any one of claims 13 to 15, characterized in that the at least one control command expresses at least one of: a navigation direction (RW, DW, LW, 130d) for a marker symbol (231, 331, 431, 531, 631) with respect to the screen (150), a rotation of at least one screen object representation (740) of at least one GUI component on the screen (150), a color change of at least one screen representation (840) of at least one GUI on-screen component (150), and a texture modification of at least one screen representation of at least one on-screen GUI component (150). The method according to claim 16, characterized in that at least one control command expresses a navigation direction (RW, DW, LW, 130d) of a cursor symbol (231, 331, 431, 531, 631) with respect to the screen (150) and the method comprises: distinguishing a navigation direction (RW, DW, LW, 130d) from a first sagged relative activation zone (430, 530), and moving a cursor symbol (231, 331, 631) across the screen (150) according to with the navigation direction (RW, DW, LW, 130d) The method according to claim 17, characterized by presenting a set of control objects (220A, 220B, 22OC, 22OD; 420A, 420B, 420C, 420D; 520A; 620) on the screen (150), each control object in the set being adapted to indicate a respective navigation direction (RW, DW, LW, 130d) of a cursor symbol (231, 331, 431, 531, 631) relative to an information area (240, 340, 440, 540, 640). The method according to claim 18, characterized by: presenting the set of control objects (22OA, 220B, 22OC, 220D) on the screen (150) in such a way that said set is gradually moved across the screen (150) according to the navigation direction (RW), and moving the cursor (231, 331, 431, 531, 631) across the screen (150) according to the navigation direction (RW). The method according to claim 18, characterized by: presenting the set of control objects (42OA, 420B, 420C, 420D; 620) at fixed positions on the screen (150), and moving the cursor (431, 631 ) across the screen (150) according to the navigation direction (DW, LW). The method according to any one of claims 13 to 20, characterized in that the screen (150) presents an information area (640) comprising editable symbols, and the method comprises: determining at least one added symbol based on a movement sequence of the gaze point (130) over a number screen objects (620, 621, 622) visually representing GUI components of an on-screen software keyboard (150), and adding at least one added symbol to the symbols in the information area (640). The method according to any one of claims 13 to 21, characterized in that the screen (150) presents an information area (640) comprising editable symbols, and the method comprises: examining whether the gaze point (130) meets an activation criterion with respect to an active control object ( 620A) representing an erase button, and if such a criterion is found to be fulfilled erasing a subset of symbols, the subset comprising at least one of the editable symbols. The method according to any one of claims 13 to 17, characterized in that the set of control objects comprises a single control object (520A), and the method comprises: determining a target location for a first sacked (130d) outside the activation zone (530), presenting the single control object (520A) at the target location, where the single control object (520A) includes a data manipulation window (520a), and moving the cursor symbol (531) across the screen (150) in accordance with a navigation direction (130d) expressed by the first 529 156 32 sackaden. The method according to any one of claims 13 to 23, characterized by: examining whether the gaze (130) meets a deactivation criterion with respect to the control object (220A; 420B; 520A; 620A, 720B, 8208), and if such a criterion exists be disabled deactivation of the active control object (220A; 420B; 520A; 620A, 7208, 820B). A computer program directly loadable into the internal memory of a computer comprising software for controlling the steps according to any one of claims 13 to 24 when said program is run on the computer. A computer readable medium (146) having a program stored thereon, the program being adapted to cause a computer to control the steps of any of claims 13 to 24.