JP2006309448A - User interface device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a user interface device and method, capable of operating or preparing for operation while estimating a user's operation intention in advance. <P>SOLUTION: A sensor part 21 regularly continues to sense the user's state even when the user does not operate a mouse 11, and supplies the sensed information to an operation intention estimation part 22. The estimation part 22 estimates the user's operation intention based on the multidimentional sensed information in advance, and supplies information related to the estimated operation intention to an operator selection part 23. The operating element selection part 23 selects an operating element corresponding to the estimated operation intention and reports the selected operating element in response to an inquiry from a display control part 12. The display control part 12 fits, upon receipt of the report of the operating element from the selection part 23, the position of a mouse cursor displayed on a display 13 to the position of the reported operating element. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a user interface device and a method thereof that can perform an operation or prepare for an operation by estimating a user's operation intention in advance.

  A user interface device that can be operated by a user normally operates in accordance with a physical operation using a user's hand, such as a pointing operation using a pointing device. However, it is difficult for a person with severe disabilities such as paralysis of the limbs to adjust the display position of the pointing device to an operator such as an icon on the display.

  Therefore, conventionally, it has been proposed to use the user's line-of-sight information for a pointing operation using a mouse which is one of pointing devices. Specifically, the following Non-Patent Document 1 proposes an Auto method in which a pointing operation is completely performed by a user's line of sight, a Manual method in which a pointing device is switched to a mouse after performing a rough pointing by a line of sight. Further, in Patent Document 1 below, it is proposed to use a user's line-of-sight information for a pointing operation, and to have a suction function that draws the pointing device to the operator between the operator closest to the display position of the pointing device. Has been. According to the user interface device provided with such a pointing device, the user basically only needs to move his / her line of sight when performing the pointing operation, so that the operation burden on the user can be reduced.

Masatake Yamato, “Gaze / Mouse Combination Target Selection Method Suitable for General GUI”, Journal of Information Processing Society of Japan, Vol.42, No.06 JP 2000-47823 A

  However, in the Auto method in Non-Patent Document 1, it takes a long time to select icons, menus, etc. due to the accuracy of gaze recognition. Even in the Manual method, the line of sight is not changed unless the line of sight is moved. Since the location cannot be calculated, it took some time until the display position of the pointing device, that is, the position of the mouse cursor initially moved. In addition, although the pointing device of Patent Document 1 has an adsorption function, it takes some time until the position of the mouse cursor actually moves.

  In addition, with any conventional pointing device, the user must always direct the consciousness / intention important to the current work to the pointing operation, and every time a pointing operation is performed, the user is warned of the current work. It was.

  Such a problem is not limited to the case of a user interface device provided with a pointing device such as a mouse, and the same applies to other user interface devices.

  The present invention has been proposed in view of such a conventional situation, and provides a user interface apparatus and method capable of operating or preparing for operation by estimating a user's operation intention in advance. The purpose is to do.

  In order to achieve the above-described object, a user interface device according to the present invention includes an interface unit that can be operated by a user, a processing unit that performs a predetermined process in accordance with a user operation via the interface unit, and a user A sensor means for sensing the state of the user, an operation intention estimation means for estimating the user's operation intention in advance using an inference model based on the state of the user sensed by the sensor means, and the operation intention estimation means. Control means for controlling the processing means in accordance with the estimated operation intention.

  In order to achieve the above-described object, a user interface method according to the present invention includes an interface unit that can be operated by a user, and a processing unit that performs a predetermined process in response to a user operation via the interface unit. A user interface method in a user interface device comprising: a sensing step for sensing a user state; and based on the user state sensed in the sensing step, the user's operation intention is determined using an inference model. It has the operation intention estimation process estimated in advance, and the control process which controls the said process means according to the operation intention estimated in the said operation intention estimation process.

  According to the user interface device and the method thereof according to the present invention, based on the sensed user state, the operation intention of the user is estimated in advance using an inference model, and the user operates according to the estimated operation intention. Since it is possible to prepare for the operation, it is possible to shorten the time from the user's intention to the operation. At this time, if the user of the user interface device operates or prepares for an initial state that is not so conscious, even if it malfunctions, it will not be recognized as a major problem, and the user only has convenience. You can enjoy it.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  Hereinafter, first, an example in which the present invention is applied to a user interface device including a display and a mouse as a pointing device will be described. As shown in FIG. 1, the user interface device 1 is roughly divided into an interface unit 10 and a user intention estimation module unit 20. Among these, the interface unit 10 includes a mouse 11, a display control unit 12, and a display 13. The user intention estimation module unit 20 includes a sensor unit 21, an operation intention estimation unit 22, and an operator selection unit 23. And have.

  In such a user interface device 1, when the user operates the mouse 11 to move the mouse cursor displayed on the display 13 to a desired position, an operation signal corresponding to the operation is sent from the mouse 11 to the display control unit 12. Supplied. The display control unit 12 changes the position of the mouse cursor displayed on the display 13 based on the operation signal. Further, when the user clicks the mouse 11 with the position of the mouse cursor set to an operator such as an icon, an operation signal corresponding to the operation is supplied from the mouse 11 to the display control unit 12. The display control unit 12 changes the display content of the display 13 based on the operation signal.

  On the other hand, even when the user is not operating the mouse 11, the sensor unit 21 continuously senses the user's state and supplies the sensed information to the operation intention estimation unit 22. As the sensor unit 21, for example, various devices capable of sensing a user's state, such as a CCD (Charge Coupled Device) camera, an infrared camera, a microphone, a sweat detector, a line-of-sight detector, and a temperature detector can be used. Those that do not require the user to wear a special device are preferred. In addition, examples of information to be sensed include those shown in Table 1 below, but information that is generated even if the user does not intend is particularly preferable. In Table 1, “settlement” can be determined from, for example, the amount of movement of the user's face, and “degree of enthusiasm” can be determined from, for example, the distance between the user's face and the display 13.

  The operation intention estimation unit 22 estimates a user's operation intention in advance based on the sensed multidimensional information, and supplies information related to the estimated operation intention to the operator selection unit 23. The operation element selection unit 23 selects an operation element corresponding to the estimated operation intention, and notifies the selected operation element in response to the inquiry from the display control unit 12 described above. When the operator is notified from the operator selector 23, the display controller 12 adjusts the position of the mouse cursor displayed on the display 13 to the position of the notified operator.

  Here, the operator in the user interface device 1 is a place where the mouse 11 can be clicked on the display 13 and the operator's operation intention is associated with each operator. For example, in the case of a window displayed on the display 13, there are operation elements 30 to 38 as shown in FIG. 2, and the operation element 30 has the operation intention to minimize the window, and the operation element 31 maximizes the window. The operation element 32 corresponds to the operation intention that the user wants to close the window (close). In addition, the operators 33 to 36 correspond to the operation intention of scrolling the display area up, down, left, and right, respectively, and the operators 37 and 38 slide the display area up and down and left and right, respectively. It corresponds to the operation intention of wanting.

  The operation intention estimation unit 22 described above receives multidimensional information sensed by the sensor unit 21 as input, and estimates a user's operation intention using an inference model. As the inference model, for example, a Bayesian network, a dynamic Bayesian network, a support vector machine, boosting, a hidden Markov model, a neural network, or the like can be used. Also, a deterministic model such as “if-then” can be used. In other words, any inference model that can select one operation intention having high correlation with sensed multidimensional information may be used.

  Hereinafter, a case where a Bayesian network is used as an inference model in the operation intention estimation unit 22 will be described.

  In order to construct a network structure of a Bayesian network, first, a node of “operation intention” and a node capable of sensing that seems to have a correlation with the operation intention are determined, and the state of each node is defined. For example, as shown in FIG. 3, in addition to the “operation intention” node, “sweat amount”, “place of hand”, and “degree of enthusiasm” nodes can be defined. Among these, as the state of the “operation intention” node, the type of each operation element such as “maximize”, “minimize”, “close”, “up scroll”, “down scroll”, and the like is defined. In addition, as the state of the “sweat amount” node, for example, “high”, “low”, and “no” are defined, and as the “hand place” node state, for example, “mouse”, “face” ”And“ keyboard ”are defined, and the state of“ heated ”, for example,“ heated ”,“ ordinary ”, and“ bored ”are defined as the state of the node.

  Subsequently, such nodes are connected to define, for example, a network structure as shown in FIG. This network structure may be automatically defined from learning data using, for example, the K2 algorithm, or may be defined from knowledge in consideration of the causal relationship. Thereafter, as shown in FIG. 5, for example, parameters of each node are defined. This parameter may be automatically defined from learning data or may be defined from knowledge.

  After the network structure of the Bayesian network is constructed in this way, when multidimensional information sensed by the sensor unit 21 is input to the operation intention estimation unit 22, the operation intention estimation unit 22 receives the input multidimensional information. The user's operation intention is estimated based on the information. For example, as shown in FIG. 6A, the information “the amount of sweating” is “large”, “the degree of enthusiasm” is “getting bored”, and “the place of the hand” is “face” is input to the operation intention estimation unit 22. And the operation intention estimation part 22 produces | generates the value about each operation intention, as shown to the same figure (B). As a result, since the value of “close” is the largest, the operation intention estimation unit 22 estimates that the user's operation intention is “close” and supplies information related to “close” to the operator selection unit 23.

  Thereafter, the operator selector 23 selects an operator corresponding to “close”, for example, the operator 32 in FIG. 2, and notifies information related to the operator 32 in response to an inquiry from the display controller 12. When the information regarding the operation element 32 is notified from the operation element selection unit 23, the display control unit 12 adjusts the position of the mouse cursor displayed on the display 13 to the position of the operation element 32.

  Thus, in the user interface device 1 described above, the user's operation intention is estimated in advance from the sensed user state, and the position of the mouse cursor displayed on the display 13 is set to the position of the operator corresponding to the operation intention. Since they are matched, the user can determine the position of the mouse cursor without being restricted not only by the line of sight but also by consciousness and intention. Therefore, the user can continue the current work even while moving the mouse cursor to a desired position, and it is only necessary to take away the line of sight, intention, and consciousness only at the moment of confirmation of the operation element.

  In the above description, the position of the mouse cursor is aligned with the position of the operator corresponding to the estimated operation intention. However, the present invention is not limited to this, and the current mouse cursor is displayed. The mouse cursor may be moved between the position and the position of the operator corresponding to the estimated operation intention.

  In the user interface device 1, the mouse cursor may not be displayed on the display when the user is not operating the mouse. In this case, the user's operation intention is estimated while the mouse cursor is not displayed on the display, and the mouse cursor is positioned at the position of the operator corresponding to the estimated operation intention at the moment when the user slightly moves the mouse. Will be displayed together. Therefore, even if the operation intention is incorrectly estimated and the mouse cursor is displayed at a position different from the user's operation intention, the user does not care about the initial position of the mouse cursor in the first place. It is not recognized as a malfunction.

  Next, an example in which the present invention is applied to a user interface device as an audio playback device including a tuner, a CD (Compact Disk) playback unit, and an MD (Mini-Disk: trademark) playback unit will be described. As shown in FIG. 7, the user interface device 2 is roughly divided into an interface unit 40 and a user intention estimation module unit 50. Among these, the interface unit 40 includes an operation button 41 and an operation control unit 42, and the user intention estimation module unit 50 includes a sensor unit 51, an operation intention estimation unit 52, and an operator selection unit 53. Have.

  In such a user interface device 2, when the user presses the operation button 41, an operation signal corresponding to the operation is supplied from the operation button 41 to the operation control unit 42. Based on the operation signal, the operation control unit 42 reproduces a radio program or reproduces a CD or MD (trademark).

  On the other hand, even when the user does not operate the operation button 41, the sensor unit 51 continuously senses the user's state and supplies the sensed information to the operation intention estimation unit 52. As the sensor unit 51, as described above, various devices that can sense the user's state, such as a CCD camera, an infrared camera, a microphone, a sweat detector, a line-of-sight detector, and a temperature detector, can be used.

  The operation intention estimation unit 52 estimates the user's operation intention based on the sensed multidimensional information, and supplies information related to the estimated operation intention to the operator selection unit 53. The operation element selection unit 53 selects an operation element corresponding to the estimated operation intention, and notifies the selected operation element in response to the inquiry from the operation control unit 42 described above. When the operation controller 42 is notified of the operation element from the operation element selector 53, the operation control unit 42 performs control according to the operation element.

  Here, the operator in the user interface device 2 is a concept close to the operation button 41 described above, and each operator is associated with the user's operation intention. For example, in the case of an audio playback apparatus, there are operation buttons 60 to 66 as shown in FIG. 8, the operation button 60 is intended to be turned on, and the operation button 61 is intended to be adjusted to adjust the volume. The operation button 62 is intended to stop playback of MD (trademark), the operation button 63 is intended to play MD (trademark), and the operation button 64 is intended to select music. The operation buttons 65 correspond to the operation intentions for extracting the MD (trademark). These operation buttons 60 to 65 can be grasped as operators. On the other hand, the operation button 66 is for selecting a mode. Depending on the number of pressing operations, the operation button 66 may be set to the MD (trademark) playback mode, the CD playback mode, the radio mode, or the AUX mode. Correspond to multiple operation intentions that you want to do. The operation button 66 can grasp each mode such as MD (trademark) reproduction mode, CD reproduction mode, radio mode, and AUX mode as an operator.

  The operation intention estimation unit 52 described above receives multidimensional information sensed by the sensor unit 41 as input, and estimates the user's operation intention using an inference model. As the inference model, as described above, an inference model that can select one operation intention highly correlated with sensed multidimensional information, such as a Bayesian network, can be used.

  Hereinafter, a case where a Bayesian network is used as an inference model in the operation intention estimation unit 52 will be described.

  In order to construct a network structure of a Bayesian network, first, a node of “operation intention” and a node capable of sensing that seems to have a correlation with the operation intention are determined, and the state of each node is defined. For example, as shown in FIG. 9, in addition to the “operation intention” node, “operation time”, “movement of hand”, and “environment sound spectrum” nodes can be defined. Among these, as the state of the “operation intention” node, the type of each operation element such as “MD playback mode”, “CD playback mode”, “radio mode”, “AUX mode”, etc. is defined. In addition, for example, “morning”, “daytime”, “night”, and “late night” are defined as the “operation time” node state, and the “hand movement” node state is, for example, “ Define the states such as “approaching with the back of the hand up”, “approaching with the thumb up”, “not approaching”, and the node status of the “environmental sound spectrum” can be, for example, “high range”, “middle range” ”And“ low range ”are defined.

  Subsequently, such nodes are connected to define, for example, a network structure as shown in FIG. This network structure may be automatically defined from learning data using, for example, the K2 algorithm, or may be defined from knowledge in consideration of the causal relationship. After that, as shown in FIG. 11, for example, parameters of each node are defined. This parameter may be automatically defined from learning data or may be defined from knowledge.

  When the multidimensional information sensed by the sensor unit 51 is input to the operation intention estimation unit 52 after the network structure of the Bayesian network is constructed in this way, the operation intention estimation unit 52 receives the input multidimensional information. The user's operation intention is estimated based on the information. For example, as shown in FIG. 12A, the “operation time” is “night”, the “movement of the hand” is “approaching with the back of the hand up”, and the “environment sound spectrum” is information “middle range”. When input to the intention estimation unit 52, the operation intention estimation unit 52 generates a value for each operation intention as shown in FIG. As a result, since the value of “CD playback mode” is the largest, the operation intention estimation unit 52 estimates the user's operation intention as “CD playback mode”, and sends information related to “CD playback mode” to the operator selection unit 53. Supply.

  Thereafter, the operator selection unit 23 selects an operator corresponding to the “CD playback mode”, and notifies information related to the operator in response to an inquiry from the operation control unit 42. The operation control unit 42 changes the mode to “CD playback mode” when information on the operation unit is notified from the operation unit selection unit 53.

  As described above, in the user interface device 2 described above, the user's operation intention is estimated in advance from the sensed user's state, and the operation is controlled according to the estimated operation intention. In this case, the CD starts to rotate before the user presses the CD playback button, and the time from the user's intention to the operation can be shortened.

  It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a figure which shows an example of schematic structure of the user interface apparatus in this Embodiment. It is a figure which shows an example of the operation element on a display. It is a figure which shows an example of the node defined in a Bayesian network, and its state. It is a figure which shows an example of the network structure which connected nodes. It is a figure which shows an example of the parameter defined about each node. It is a figure which shows the example of estimation of a user's operation intention. It is a figure which shows the other example of schematic structure of the user interface apparatus in this Embodiment. It is a figure for demonstrating the operation element in an audio reproducing apparatus. It is a figure which shows an example of the node defined in a Bayesian network, and its state. It is a figure which shows an example of the network structure which connected nodes. It is a figure which shows an example of the parameter defined about each node. It is a figure which shows the example of estimation of a user's operation intention.

Explanation of symbols

1, 2 User interface device, 10 Interface unit, 11 Mouse, 12 Display control unit, 13 Display, 20 User intention estimation module unit, 21 Sensor unit, 22 Operation intention estimation unit, 23 Operator selection unit, 40 Interface unit, 41 Operation buttons, 42 operation control unit, 50 user intention estimation module unit, 51 sensor unit, 52 operation intention estimation unit, 53 operator selection unit

Claims (4)

Interface means that can be operated by the user;
Processing means for performing predetermined processing in response to a user operation via the interface means;
Sensor means for sensing the user's condition;
Based on the state of the user sensed by the sensor means, an operation intention estimation means for estimating the user's operation intention in advance using an inference model;
And a control means for controlling the processing means in accordance with the operation intention estimated by the operation intention estimation means. The interface means has a pointing device,
The user interface device according to claim 1, wherein the control unit controls the processing unit so that a pointing position by the pointing device is a position corresponding to the estimated operation intention.   2. The user interface device according to claim 1, wherein the inference model is a Bayesian network, a dynamic Bayesian network, a support vector machine, boosting, a hidden Markov model, a neural network, or a deterministic model. A user interface method in a user interface device comprising: interface means capable of being operated by a user; and processing means for performing predetermined processing in response to a user operation via the interface means,
A sensing process for sensing the user's condition;
Based on the state of the user sensed in the sensing step, an operation intention estimation step of estimating the user's operation intention in advance using an inference model;
And a control step of controlling the processing means according to the operation intention estimated in the operation intention estimation step.

Images