JP2000250677A - Device and method for multimodal interface - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a natural and smooth interaction by presenting guide information for supporting the interaction to the user in optimum manner in a multimodal interface environment. SOLUTION: This device is provided with an attention information generating means 104 for detecting an attention spot of a user and generating it as attention information, a guide information control means 105 for finding a position to present the guide information for supporting the input of the user on the basis of the attention information and a presentation control means 101 for presenting and controlling the guide information at the position found by the guide information control means 105 and the guide information is presented near the attention spot of the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a multimodal interface device and a multimodal interface method. In particular, in a multi-modal interface environment in which data is exchanged between the user and the computer in multiple interaction modes, natural and smooth interaction is realized by optimally presenting guide information that assists the interaction to the user. Related to technology.

[0002]

2. Description of the Related Art In recent years, in various computer systems including personal computers, multimedia information such as audio information and image information has been input in addition to the conventional input and output of characters and images by a keyboard and a mouse. Output was realized.

One of the interactive systems using such multimedia information is a voice interactive system. Due to the progress of natural language analysis / natural language generation, speech recognition / speech synthesis technology, or interaction processing technology, there is an increasing demand for a speech interaction system for interacting with a user with speech input / output data. For example, "TOSBURG-II" which is a dialogue system based on voice input by free utterance (Transactions of the Institute of Electronics, Information and Communication Engineers, Vol. J77-D-II, No. 8, p.
pp. 1417-1428, 1994), etc., have been developed.

In addition to the voice input / output, visual information input data grasped using, for example, a camera is used, or a touch panel, a pen, a tablet, a data glove, a foot switch, an interpersonal sensor, a head mounted display, and a force display are provided. By using information that can be exchanged with the user through various external input / output devices such as a power device, there is an increasing demand for a multi-modal interaction system for interacting with the user. Hereinafter, the user interface having the plurality of interactive modes is referred to as a multimodal interface (MMI).

[0005] In a dialogue between humans, communication is not performed using only one medium (channel) such as voice, but gestures, gestures, and gestures are performed.
By interacting with non-verbal messages transmitted and received through various media such as facial expressions, natural and smooth interaction is realized (“Intel”
light Multimedia Interfa
ces ", Maybury MT, Eds., The
AAAI Press / The MIT Pres
s, 1993). Similarly, in the interaction between a human and a computer, the multimodal interface is a powerful technique for realizing a natural and easy-to-use human interface.

The processing of the conventional multimodal interface will be described below.

[0007] When a user inputs a voice or the like to a computer, the input voice waveform signal is converted from analog to digital. The voice section is detected by, for example, calculating the power per unit time of the digitized voice signal. The audio signal is analyzed by a method such as FFT (Fast Fourier Transform). Next, for example, using a method such as HMM (Hidden Markov Model), the analyzed speech signal is collated with a speech recognition dictionary, which is a standard pattern prepared in advance, and the utterance content is estimated according to the collation result. Is done. Processing according to the estimated occurrence is performed.

[0008] In the case of input by a non-verbal message other than voice input, analysis of image information of a user obtained from a camera, analysis of output information such as a distance sensor using infrared rays, or contact type such as a touch sensor By inputting a non-verbal message such as a gesture from a user by recognizing the position, shape, or movement of the user's hand by analyzing the output information of the input device.

[0009]

However, the conventional multimodal interface has the following problems.

In the first place, the multimodal interface environment has the following characteristics different from the conventional one. That is,
First, it is difficult to determine whether the input information is information intended by the user or unnecessary information.

Second, since the meaning of input data is estimated by collation processing with a dictionary or the like registered in advance, guide information such as input candidates for a user is indispensable for improving the accuracy of the collation. . Hereinafter, this input guide will be described in detail.

In the current technology, the analysis accuracy of the collation processing of the input from each medium is low, and the properties of each input / output medium are not sufficiently clarified. For this reason, a multimodal interface that efficiently uses each newly available I / O media or multiple I / O media, is highly efficient, effective, and reduces the burden on users is still realized. Not.

[0013] Further, when inputting by voice or gesture, it is difficult for the user to understand the expression that is an input candidate such as a vocabulary or a gesture type that can be currently input for each scene. At the same time, it is difficult to know at what point the input is possible.

In an interface using a recognition technique such as a voice or a gesture, there is uncertainty as to whether or not the input content is input as intended because of the low recognition accuracy. For this reason, it is difficult for the user to understand whether each input has been correctly recognized.

In order to cope with these, it is necessary to appropriately present a guide for input to the user. This guide shows the expressions that can be entered now, the current state of accepting input, or the timing of input, or displays the recognition result of the entered expression on a part of the display area, etc. To assist.

In general, the combinations of expressions that can be input at each point of input of voice, gesture, and the like are complicated and enormous. Further, in many cases, the user not only inputs a predetermined expression, but also determines and changes the expression one by one on the spot and inputs the expression. By referring to the above input guide, the user does not need to remember all combinations of these expressions.

With this input guide, the user can determine an expression to be input while looking at candidates for expressions that can be input at each point in time. In addition, when performing input by voice or gesture, the input timing can be obtained. For these reasons, users often refer to this input guide when performing input.

However, the conventional input guide has the following problems.

Generally, a multimodal interface is
It is used in combination with a device or a method for performing some task, and enables input to these devices or the like using a recognition technique such as voice or gesture. Therefore, to execute the original task, the user needs to look at the work area corresponding to the task most of the time.

The area where the user is currently performing the work and the area where the above-mentioned guide is displayed are as follows:
In most cases it will be another area. When using input means using recognition technology such as voice or gesture, the user may need to look at the guide, but if the user is throwing his / her gaze only at the work area, the voice or gesture may be used. You will not be able to see the guide for input. On the other hand, when the user is throwing his / her line of sight only at the guide, the user cannot perform the original task because the work area cannot be seen. On the other hand, when the user attempts to throw his / her gaze at both the guide and the work area, the user's gaze frequently moves between them. For this reason, the user needs to move his / her gaze to realign his / her gaze in both areas, or to perform extra work to find the information currently needed in both areas.
The load on the user increases.

As described above, the first problem is that the work area and the input guide are displayed in different areas, so that a medium such as a voice or a hand gesture using a medium which is not originally restricted by eye movements is used. In spite of the use of the media, it limits the movement of the eyes of the user, and consequently negates the original merit of these media.

As a second problem, when the result of analyzing the input contents is fed back to the user, a sound signal such as a beep sound or a companion sound has conventionally been used as the feedback signal.

However, depending on the surrounding environment, always performing feedback by a voice signal causes noise to the surroundings. Or it becomes complicated for the user himself.

As a third problem, a means for controlling a dialog in an input guide between a user and a computer is required. Specifically, it is a means for detecting the occurrence of a failure when any failure relating to communication with the user occurs and eliminating the failure. These obstacles include a failure in recognizing an input from a user or a failure in outputting information to a user. In order to solve these obstacles, for example, the user must respond to the input from the user by re-presenting the information for confirmation, asking the user the question and answering the question, and appropriately controlling the flow of the discussion of the dialog. It is necessary to perform appropriate output to the user by using such a method, and to appropriately control the timing of the input from the user and the output to the user.

Various methods have been used in the conventional dialog management processing assuming input devices such as a mouse and a keyboard. For example, there is a method using a script that is a flow of a dialog prepared in advance, and a method using information such as a utterance pair and an utterance exchange structure, which are pairs of utterances such as a question / answer / greeting / greeting. . Further, a method of planning, describing, generating, and recognizing the entire flow of the dialogue as a plan (plan) of each individual of the dialogue participants or a joint plan (plan) between the participants is used.

However, especially in the case of multimodal input such as voice or gesture, the user inputs while considering the expression to be input, cancels the input midway, or takes time to determine the input content. Therefore, it is difficult to control the appropriate timing when controlling the time for receiving the input.

There is a first method of controlling this timing control so as to receive an input for a predetermined period of time. However, in this method, if the user takes a long time to input, as described above, input becomes impossible.

On the other hand, there is a second method in which an input is always accepted in consideration of a possibility that a user performs an input at an arbitrary timing. However, in this method, a voice or an operation that is not intended to be input is erroneously received, or ambient noise, an unrelated operation or image, or the like is erroneously received. For this reason, a malfunction has occurred and the burden on the user has increased.

On the other hand, there is a third method for lengthening the time for accepting an input from a user in consideration of the possibility that the user performs an input at a timing later than usual. However, similarly to the second method, this method erroneously accepts a voice or an operation that is not intended to be input, or erroneously accepts surrounding noise / irrelevant operation / image. For this reason, a malfunction has occurred and the burden on the user has increased.

Further, there is no method for determining whether or not the user intends to continue inputting. For this reason, it is not possible to determine an appropriate timing as to how long the input waiting time should be extended.

Therefore, in the conventional multimodal interface, it is possible to effectively utilize non-verbal messages such as eyes, gestures such as gestures and hand gestures, and facial expressions which play an important role in communication between humans. could not.

As described above, the present invention is used in a multimodal interface environment to improve the accuracy of recognition of input by each medium, and various guide information presented to assist a dialogue is originally used. The input contents that were presented in a different area from the application processing to be processed, and the feedback by audio signal was performed uniformly regardless of the situation, and the input contents intended by the user when the failure occurred in the dialogue etc. The purpose of the present invention is to solve the problem that the input is not always performed correctly and the burden on the user is increased, and the input guide cannot be performed properly.

The purpose is to present guide information for assisting the dialog in correspondence with the work area where the user works, thereby reducing the burden on the user and appropriately input guide. To provide a multi-modal interface device and a multi-modal interface method that enable efficient use of a plurality of input / output media.

Another object of the present invention is to provide more reliable and natural feedback of an input to a user by performing feedback by an audio signal as necessary according to the surrounding situation.

Another object of the present invention is to obtain input information intended by the user more efficiently by appropriately controlling the timing or flow of the dialogue.

[0036]

A feature of the present invention for solving the above-mentioned problem is that the guide information for assisting the user's input is optimally presented in the vicinity of the user's gaze point. In that

This gaze point is obtained, for example, from the direction of the user's line of sight.

A first feature of the present invention for realizing such a function is a multimodal interface device for inputting / outputting information between a user and a computer in a plurality of interactive modes. Gaze information generating means for detecting a gaze point of the user and generating it as gaze information, based on the gaze information, guide information control means for determining a position to present guide information for assisting the input of the user, The present invention is characterized in that there is provided a presentation control means for controlling the presentation of guide information at the position determined by the guide information control means.

According to the above configuration, guide information can be presented near the user's work area. Accordingly, the guide information can be surely recognized by the user without increasing the burden on the user.

According to a second feature of the present invention, the guide information control means further includes a predetermined distance between the user's gaze point indicated by the gaze information and a presentation position of the guide information. A first judging means for judging whether or not the distance is within a range of a first threshold value, and when the distance is not within the range of the first threshold value, the guide is located near a gaze point of the user. The point is to set the information presentation position.

According to the above configuration, guide information can be presented near the work area of the user. Accordingly, the guide information can be surely recognized by the user without increasing the burden on the user.

A third feature of the present invention is that the guide information control means further determines a presentation position of the guide information when it is determined that the user is paying attention to the guide information. The point is that it is fixed.

According to the above configuration, when the user has already captured the guide information, the guide information is not moved, so that unnecessary flicker can be eliminated and the burden on the user can be reduced. .

A fourth feature of the present invention is that the guide information control means further includes a step of determining whether or not the determined presentation position of the guide information is within a predetermined presentation area in which the guide information is to be presented. A second determining means for determining whether
When the present position is not in the present area, the present position of the guide information is corrected in the present area.

According to the above configuration, when the calculated guide information presentation position exceeds the display screen or the like, the presentation position can be automatically offset. Therefore, it is possible for the user to reliably recognize the guide information.

A fifth feature of the present invention resides in that the guide information control means corrects the size of the guide information so as to present the guide information within the visual field of the user.

According to the above configuration, all guide information to be presented in the user's field of view can be stored. Therefore, the load on the user for acquiring guide information is reduced.

According to a sixth feature of the present invention, the multimodal interface device further includes at least one of usage status of each application presented in the presentation area and arrangement information of each presentation element of the application. The present invention is characterized in that it comprises application status grasping means for generating application presentation information indicating the application information, and the guide information control means sets the guide information to a position which does not hinder the presentation of each application based on the application presentation information. .

According to the above configuration, guide information can be presented without obstructing information displayed by each application on the screen. Therefore, the efficiency of interaction with each application is improved.

A seventh feature of the present invention is that the multi-modal interface device further comprises an auxiliary voice generating means for generating voice guide information by a voice signal for assisting an input from a user; Output control means for controlling the output of voice guide information, the guide information control means, based on the gaze information, when it is determined that the user can not capture the guide information in the vicinity of the visual field region, The point is that the voice guide information is set as guide information to be presented.

According to the above configuration, even when the user is not watching the display screen, the user can recognize the guide information.

According to an eighth feature of the present invention, the multimodal interface device is further configured such that, based on the gaze information, the gaze point of the user stays in an area of the guide information in which the gaze point is presented. The guide information control means, when it is determined that the gaze point of the user is stagnant, the control means, from the user The point is to maintain the state of waiting for the input of.

According to the above configuration, the waiting time for input from the user can be optimized. Therefore, the load on the system is reduced.

A ninth feature of the present invention resides in that the presentation control means controls the presentation of the guide information in a presentation area by translucent display or emphasized display.

According to the above configuration, guide information can be presented without obstructing already displayed display elements.

Further, a tenth feature of the present invention is a multimodal interface method for inputting and outputting information between a user and a computer in a plurality of interactive modes, and detects a gazing point of the user. And generating a gaze information, based on the gaze information, determining a presentation position to present guide information for assisting the user input, and presenting the guide information at the presentation position Is included.

According to the above configuration, guide information can be presented near the user's work area. Accordingly, the guide information can be surely recognized by the user without increasing the burden on the user.

An eleventh feature of the present invention is a computer-readable recording medium storing a multimodal interface program for inputting and outputting information between a user and a computer in a plurality of interactive modes. A module for detecting a gaze point of the user and generating it as gaze information; a module for determining a presentation position for presenting guide information for assisting the user's input based on the gaze information; And a module for presenting guide information at a position.

According to the above configuration, guide information can be presented near the user's work area. Accordingly, the guide information can be surely recognized by the user without increasing the burden on the user.

[0060]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings. The first embodiment provides a function of controlling the position of the input guide information according to the position of the user's line of sight.

FIG. 1 is a block diagram showing a functional configuration of the multimodal interface device according to the first embodiment of the present invention. As shown in FIG. 1, the multimodal interface device 1 according to the first embodiment includes an output unit 1
01, an input unit 102, a visual guide presenting unit 103,
It comprises a gaze target detecting unit 104 and a control unit 105.

The output unit 101 outputs data from various media to the user from inside the computer.
The output unit 101 outputs at least visual information to a user such as a CRT display, an LCD display, a projection projector, and a head-mounted display. For example, when outputting with a GUI (graphical user interface), for example, a window, a menu, a pointer, and the like are displayed on a bitmap display.

In FIG. 1, reference numeral 102 denotes an input unit.

The input section 102 inputs data from a user to a computer by various media. The input unit 102 includes at least one of a microphone, a camera, a keyboard, a pointing device such as a touch panel, a pen, a tablet, a mouse, and a trackball, a data glove, a data suit, an eye tracker, a head tracker, an OCR, an interpersonal sensor, and a seating sensor. It takes in input of voice information, visual information, operation information, etc. from the user obtained through the input device. Speech information, visual information, and operation information input are sampled, coded, digitized, filtered, signal converted, recorded, saved, pattern-recognized, analyzed for language / speech / image / operation / operation, understanding, and intention extraction The input information is analyzed by performing at least one process.

The visual guide presenting unit 103 includes the input unit 102
Is presented to the user as guide information, which is information for assisting the user to input from the user. The visual guide presenting unit 103 displays the guide information on the output unit 101 by visual means such as characters and images. In this display, the guide information may be displayed so as to overlap other display elements such as a window, a menu, and a pointer. Alternatively, it may be presented in a display form in which display attributes such as color, font, blink, highlight, and the like are changed, which are translucent and superimposed.

The contents of the guide information to be presented are, for example, firstly, whether input is accepted or not at each point in time, a list of vocabulary that can be input when voice input is assumed, or input when gesture input is assumed. There is information on candidate expressions that can be currently input by the user, such as a list of possible gesture type names and symbols or pictorial expressions. Second, there is information on the progress of processing of input from the user or information on recognition candidates obtained by processing of input from the user. The information is configured to be appropriately presented as information to assist the user in inputting.

Further, the visual guide presenting unit 103 presents guide information at a designated position on the output unit 101 in a designated format under the control of the control unit 105 described later.
Note that the visual guide presentation unit 103 corresponds to a presentation control unit in the claims.

In the following, in the first embodiment,
As input information from the input unit 102, a voice recognition input is assumed. As the guide information presented by the visual guide presentation unit 103, a speech recognition vocabulary that is a candidate for utterance that can be input at each time point is appropriately presented in a translucent manner.
However, the display contents and the display format in the visual guide presentation unit 103 are not limited, and it goes without saying that any method can be used.

The gaze target detection unit 104 determines whether or not the user is currently viewing the computer screen or the like, and at least one of the place, coordinates, area, direction, object, or part thereof to which the user is looking. Detect one. The detected information is output as gaze target information. Specifically, the gaze target information includes, for example, an eye tracker device for observing the user's eye movement, a head tracker device for detecting the movement of the user's head, a seating sensor, and Japanese Patent Application No. 08-05.
9071 A method for detecting a user's line of sight by processing image information obtained from a camera for observing the user or a camera worn by the user using the method used in the “gaze point estimation device and method”, and the like. Is detected by
The gaze target detecting unit 104 corresponds to a gaze information generating unit in the claims.

The control unit 105 includes the output unit 101 and the input unit 1
02, visual guide presenting unit 103, gaze target detecting unit 104
Controls and manages each of the components. The control unit 105
Corresponds to the guide information control means in the claims.

Next, the hardware configuration of the multimodal interface device according to the first embodiment will be described. The multimodal interface device 1 according to the present invention is a so-called general-purpose computer, workstation, P
C, a computer such as a network terminal, or a computer in which various computers are interconnected. Alternatively, the present invention may be implemented in a face-to-face device system such as a vending machine, a ticket vending machine, and a game machine.

FIG. 2 shows an example of the internal configuration when the first embodiment is mounted on a general-purpose computer. The general-purpose computer illustrated in FIG. 2 includes a CPU unit 501, a memory unit 502, a large-capacity storage unit 503, and a communication interface unit 504. The general-purpose computer shown in FIG. 2 further includes input interface units 505a to 505n and an input device unit 506a.
To 506n and output interface units 507a to 507
m, and output device units 508a to 508m.
Note that the memory unit 502 and the large-capacity storage unit 503 may be shared.

The input device units 506a to 506n are mounted on, for example, a microphone, keyboard, pen tablet, OCR, mouse, switch, touch panel, camera, data glove, data suit, and the like. Output device section 50
8a to 508m are mounted on, for example, a display, a speaker, a force display, and the like. CPU unit 501
Implements the functions of the first embodiment by controlling software that implements the multimodal interface device and method.

Note that programs for implementing various processes of the multi-modal interface of the present invention can be stored in various recording media. By reading out such a recording medium by the CPU unit 501 in the general-purpose machine having the above hardware and executing the program,
The present invention is implemented. Here, the recording medium is, for example,
Semiconductor memory, magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD)
Etc.) and all devices capable of recording programs. Further, the program may be provided through various communication means such as a network.

The first embodiment of the present invention is configured as described above, and the flow of the processing will be described below in order with reference to FIGS.

The control unit 105 controls each attribute such as the presentation position and the presentation size of the visual guide presentation unit 103 according to the following procedure.

First, the presentation position determination processing of the visual guide presentation unit 103 will be described.

FIG. 3 is a flowchart showing a procedure of the process of determining the presentation position of the visual guide presentation unit performed by the control unit 105.

Step S10 is the same as the visual guide presentation section 10
The value L0 of the normal display position set in advance is set in the register L holding the display position of No. 3.

In step S20, the utterance candidate W0 in the preset initial state is set in the register W for holding the utterance candidate.

In step S30, the visual guide presenting unit 103 is displayed on the output unit 101 in a semi-transparent manner at a position centered on the coordinates indicated by the register L.

At step S40, the gaze target detecting unit 104
It is determined whether or not the user's gaze position E has been obtained from.
When the gaze position E is obtained, the process proceeds to step S60. On the other hand, if the gaze position E cannot be obtained, the process proceeds to step S50.
Proceed to.

A step S50 decides whether or not a new set of utterance candidates Wi has been obtained. When a new set of utterance candidates Wi is obtained, the process proceeds to step S80. on the other hand,
If there is no change in the utterance candidate set W0, the process proceeds to step S
Return to 40.

A step S60 compares the contents of the register L with the contents of the gaze position E. As a result of the comparison, if the difference between the center coordinates of both the register L and the gaze position E is equal to or smaller than the threshold value F1 determined in advance based on a general user's peripheral visual field range, the process proceeds to step S50. on the other hand,
If the deviation of the center coordinates is larger than the threshold value F1, the process proceeds to step S70.

In step S70, the contents of register L are stored in E
And the process proceeds to step S100.

In step S80, the content of the register W is changed to W
Update with i.

Step S90 is the same as that of the visual guide presentation unit 10
3 is set to the contents of the register W.

In step S100, the contents of register M are determined by the presentation correction process. Details of the presentation correction process will be described later. The register M is a register that holds a presentation position used in the presentation correction process.

In step S110, the output unit 101
Visual guide presentation unit 1 centering on the coordinates indicated by register M
03 is displayed translucently. After this processing, step S4
Return to 0.

Next, the presentation correction processing of the visual guide presentation unit 103 will be described.

FIG. 4 is a flowchart showing the procedure of the presentation correction process of the visual guide presentation unit performed by the control unit 105.

A step S101 copies the contents of the register L to the register M.

In step S102, the size of the display area required for presentation by the visual guide presentation unit 103 when displaying the current vocabulary W is defined in advance based on the upper limit range of the peripheral vision of a general user. It is determined whether it is equal to or smaller than a threshold value F2 representing the size of the area. If the required size of the display area is equal to or smaller than the threshold F2, step S104
Proceed to. On the other hand, if the required display area size is larger than the threshold F2, the process proceeds to step S103.

Step S103 is the visual guide presentation section 1
03 is adjusted so that the required display area size is equal to or smaller than the threshold value F2. The adjustment of the size can be performed by changing the display style such as reducing the display font of the visual guide presenting unit 103, for example.

In step S104, it is determined whether or not the entire display area of the visual guide presenting unit 103 fits in the output unit 101, centering on the contents of the register M. If the entire display area of the visual guide presenting unit 103 fits in the output unit 101, the process ends. On the other hand, if the entire display area of the visual guide presenting unit 103 does not fit in the output 101, the process proceeds to step S105.

In step S105, the contents of the register M are shifted by a predetermined distance D toward the center of the output unit 101 to update the contents of the register M.

In step S106, it is determined whether or not the difference between the center coordinates of the register M and the register L exceeds a predetermined threshold value F3 representing the upper limit range of the peripheral vision of a general user. If the amount of deviation between the center coordinates of the register M and the register L exceeds the threshold value F3, a value L0 of a preset normal display position is set in the register M,
The presentation correction processing ends. If the shift amount between the center coordinates of the register M and the register L is within the range of the threshold value F3, the process returns to step S104.

The range of the peripheral visual field of the user represents, for example, an area in which the user can confirm the display content without moving the visual field position.

In the first embodiment, the threshold value F
It is assumed that 1, F2, and F3 have an arbitrary size in an area sufficiently smaller than the entire output unit 101.

Next, a specific example of the operation of the first embodiment according to the above processing will be described in detail with reference to FIG.

Here, it is assumed that the upper right position of the output unit 101 is designated as L0, which is the initial position of the input guide presenting unit. It is also assumed that [“w1”, “w2”, “w3”] is set in the utterance candidate set W0 in the initial state.

First, steps S10 and S2 shown in FIG.
0 and S30 are executed, and the initial presentation position L0 and the utterance candidate W0 of the input guide are set, respectively. FIG.
As shown in (a), the current recognition candidate is translucently displayed as guide information at the upper right of the display screen as the output unit 101.

Here, it is assumed that the user gazes around the vicinity (E1) of the current guide information as shown in FIG. 5B.

In this case, steps S40 and S40 shown in FIG.
Step 60 is executed, and it is confirmed that the user can now check the guide information in the peripheral vision. Therefore, the guide information is displayed as it is. The user can input, for example, “w1” while confirming the vocabulary that can be input in the peripheral vision.

In this state, the user turns his / her gaze near the gaze position E1 in order to see the contents of the input guide or to check other information displayed in the vicinity. Suppose. Even in the case where the line of sight is moved in the vicinity, the input guide continues to be displayed at a fixed position without being moved by the process of step S60 in FIG. Through these processes, an interface that does not increase the burden on the user due to, for example, the input guide frequently moving as the user's line of sight moves is realized.

Next, as shown in FIG. 5 (c), when the user shifts his / her gaze to around the gaze position E2 in order to refer to or operate the application A1 located away from the current input guide. Is assumed.

This movement of the line of sight is determined by the gaze target detecting unit 103.
And notifies the control unit 105. The control unit 105 determines that the line-of-sight position E2 is separated from the value of the register L indicating the present position of the input guide by the threshold value F1 or more. Steps S40 to S7 shown in FIG.
The process of 0 determines that the presentation position of the input guide should be changed.

According to this determination result, as shown in FIG. 5D, step S100 in FIG. 3 (step S10 in FIG. 4) is performed.
In the processing from 1 to S107), the input guide is automatically moved to a position where the user can confirm it in the peripheral visual field.

Note that the input guide is displayed translucently. This translucent display does not hide the display of the application A2 currently being searched or referenced by the user. For this reason, the user can perform input, operation, reference, and use of the application without any trouble.

Next, as shown in FIG. 5E, there are cases where the display size of the input guide exceeds the peripheral visual field due to the large number of utterance candidates to be presented. In this case, the process of step S103 in FIG. 4 reduces the display size of the input guide. For this reason, the user can put the information of the input guide such as the utterance candidate in the field of view.

Furthermore, as shown in FIG. 5F, the user's gaze position E3 may be near the periphery of the output unit 101, and the calculated presentation position of the input guide may protrude from the output unit 101. In this case, steps S104 to S104 in FIG.
In the process of step S107, the display position of the input guide is appropriately offset toward the center of the output unit 101. For this reason, the user can put the information of the input guide in the field of view without any trouble.

If the recognizable utterance candidates are changed at all points in time, the processing of steps S40 to S110 in FIG. 3 sequentially updates the display contents of the input guide. At the same time, when the required size change of the display area is changed due to the change of the display content, the display position and the display format are appropriately changed.

In the first embodiment, a speech recognition input is taken as an example of the input unit 102, but the input means is not limited to this. For example, the first embodiment is applicable to a gesture input.

In the first embodiment, a device having one display is used as the visual output unit 101, but the output unit 101 is not limited to this. For example, the first embodiment can be used in a multi-monitor environment having a plurality of displays, or in a virtual space environment using, for example, a head-mounted head-mounted display.

Further, in the first embodiment, an example in which the display mode, the display position, the size, and the like are controlled is described as a method of controlling the visual guide presenting unit 103, but the control method is not limited to this. For example, it is possible to control the output form of the visual guide presenting unit to be deformed, a plurality of outputs to be arranged, or to be displayed around the gaze position of the user.

Further, in the first embodiment, a large amount of information is presented to the visual guide presentation unit 103 by reducing the display, but the presentation method is not limited to this. For example, it is also possible to use a display format that automatically scrolls.

Further, the visual guide presenting unit 103 can be displayed only during a period of receiving and receiving an input such as a voice or a gesture.

The output of the visual guide presenting unit 103 is
Only during a period of receiving and receiving an input of a voice, a gesture, or the like, a display can be made to follow the user's line of sight.

Further, in the first embodiment, the distance from the gaze position is used to determine the peripheral visual field of the user, but the determination method is not limited to this. For example, it is also possible to additionally use another criterion in consideration of the characteristics of the human visual field.

Further, the functions of the above-described first embodiment can be used in appropriate combinations. For example, when the user is looking at a certain position, it is a nearby position where the user can check the contents without greatly moving the line of sight, and the display element on the screen that the current user is using and referring to It is possible to present the input guide information at a position where the input guide information does not overlap with or minimizes the overlap.

According to the first embodiment, the following effects can be obtained.

The gaze target detecting unit 104 detects gaze information based on the detected gaze position of the user. Control unit 10
5 controls the input guide to be displayed in the vicinity of the gazing point of the user based on the gaze information and the current display position of the input guide.

Therefore, the user who works can keep the input guide within the peripheral visual field without any trouble. Therefore, the user can efficiently grasp the contents to be input.

Second Embodiment Hereinafter, a second embodiment of the present invention will be described in detail with reference to the drawings, only with respect to differences from the first embodiment.

The second embodiment provides, in addition to the first embodiment, a function for avoiding a collision between processing of an application running on a computer and an input guide. FIG. 6 is a block diagram illustrating a functional configuration of the multimodal interface device according to the second embodiment of the present invention. In the second embodiment, an output unit 101 and an input unit 1
02, the visual guide presenting unit 103, and the gaze target detecting unit 1
04, the control unit 206, and the application management unit 20
6 is provided.

The application management unit 206 arranges the display elements, such as window menus, which are displayed on the output unit 101 at each point in time.
The display state is sequentially managed. This state of the application can be grasped by using a generally known technique for monitoring the state of a task related to the application. The application management unit 206 controls the control unit 20
5 in response to an inquiry from the client 5 or asynchronously with the control unit 205. still,
The application management unit 206 corresponds to an application status grasping unit in the claims.

The control unit 205 has almost the same function as the control unit 105 of the first embodiment. However, the control unit 205
Performs the following processing instead of step S104 in FIG. That is, the control unit 205 determines whether or not the entire display area of the visual guide presenting unit 103 fits in the output unit 101, centering on the contents of the register M. In the second embodiment, the control unit 205 further refers to the application management unit 206, and the entire display area of the visual guide presenting unit 103 centering on the contents of the register M is used by another application that runs. It is determined whether or not the display element overlaps. If the input guide and another display element overlap, the current position of the input guide is moved to a position separated from the other display element. On the other hand, if they do not overlap, the process ends.

The other configuration and processing procedure of the second embodiment are the same as those of the first embodiment, and thus the description is omitted.

In the second embodiment, the control unit 205 uses the application management unit 206 to present guide information while avoiding an area having another display element. However, the display method is not limited to this. . For example, by analyzing the movement of the user's line of sight and the operating state of other input / output elements,
Even if the application currently used by the user or the display element such as the application or window currently referred to by the user is identified, the input guide may be rearranged so as to avoid duplication with these elements. Good.

According to the second embodiment, the following effects can be further obtained in addition to the effects of the first embodiment.

The application management unit 206 monitors the status of the running application. Control unit 205
Corrects the position of the input guide appropriately according to the display state of the application. For this reason, overlapping display of the visual guide information and other display elements is avoided, and it is not necessary for the user to interrupt the application processing for the interaction via the input guide. Therefore, the work efficiency of the user is improved, and an interface that is easier for the user to understand is realized.

Third Embodiment Hereinafter, a third embodiment of the present invention will be described in detail with reference to the drawings, only with respect to differences from the first embodiment and the second embodiment.

In the third embodiment, in addition to the first and second embodiments, a function of supplementing the output to the user with auxiliary voice is provided.

FIG. 7 is a block diagram showing a functional configuration of the multimodal interface device according to the third embodiment of the present invention. The third embodiment includes an output unit 101,
The input unit 102, the visual guide presenting unit 103, the gaze target detecting unit 104, the control unit 306, and the auxiliary voice presenting unit 30
7 is provided.

[0135] The auxiliary voice presentation unit 307 performs feedback by a voice signal for the purpose of, for example, confirming that the input from the user has been correctly received or prompting the input from the user. This feedback is called auxiliary sound. The auxiliary voice is presented to the user by a buzzer, reproduction of a digitally recorded voice signal, or output of a synthesized voice. The auxiliary voice presentation unit 307 corresponds to an output control unit in the claims.

The control section 305 has almost the same function as the control section 105 of the first embodiment. However, the control unit 305
Performs the following processing.

That is, the control unit 305 uses the auxiliary voice presenting unit 307 as appropriate when the visual guide presenting unit 304 presents feedback on the input contents of the user. Specifically, the control unit 305 compares the gaze position E of the user obtained from the gaze target detection unit 103 with the contents of the register M indicating the position where the visual guide information is currently presented. If the displacement between the two positions is larger than a predetermined threshold value F4, the control unit 305 determines that the user does not include the visual guide information in the visual field. When the visual guide information is not in the visual field, the control unit 305 presents a feedback by an audio signal from the auxiliary audio presentation unit 307. Note that the control unit 305 or the auxiliary voice presentation unit 307
It corresponds to the auxiliary voice generating means in the claims.

[0138] Other configurations and processing procedures of the third embodiment are the same as those of the first and second embodiments, and a description thereof will be omitted.

In the third embodiment, an example is shown in which an audio signal output is used to assist visual guide information.
The third embodiment is not limited to this. For example, the third embodiment can be applied to other output signals as long as the output does not restrict the user's vision, such as an output using vibration or force. In addition, a companion voice often used in conversation between humans may be used as the auxiliary voice.

It goes without saying that the third embodiment may be implemented in combination with the first and second embodiments as appropriate.

According to the third embodiment, the following effects can be further obtained in addition to the effects of the first and second embodiments.

Under the control of the control unit 305, the auxiliary voice presenting unit 307 presents a feedback to the input from the user by the auxiliary voice together with or instead of the visual guide presenting unit 103. Thereby, when the user is not looking at the visual guide information, the feedback by the auxiliary voice complements the visual guide information, and when it is unnecessary, the auxiliary voice is not presented.
Therefore, an interface that more reliably provides input feedback to the user is realized.

Fourth Embodiment Hereinafter, a fourth embodiment of the present invention will be described in detail with reference to the drawings, only with respect to differences from the first to third embodiments.

The fourth embodiment further provides a function of adjusting the time for waiting for an input from a user in addition to the above-described embodiments.

FIG. 8 is a block diagram showing a functional configuration of a multimodal interface device according to a fourth embodiment of the present invention. The fourth embodiment includes an output unit 101,
The input unit 102, the visual guide presentation unit 103, the gaze target detection unit 104, the control unit 405, and the search state detection unit 40
8 is provided.

The search state detection unit 408 monitors the user's line of sight provided sequentially from the gaze target detection unit 103 according to rules prepared in advance. Specifically, for example, when the user's line of sight stays on the visual guide presenting unit 104 presenting the input candidate, the search state detection unit 408 determines the candidate to be input by the user. Analysis such as judging that selection is being performed is performed. The analysis result is transmitted to the control unit 40
5 is successively notified. Note that the search state detection unit 408 corresponds to a third determination unit in the claims.

The control section 405 has almost the same function as the control section 105 of the first embodiment. However, the control unit 405
Performs the following processing.

That is, the control unit 405 controls the search state detection unit 4
In accordance with the user's search status obtained from step 08, the input standby time is extended, the input guide information presentation time is adjusted, etc., as appropriate.

[0149] Other configurations and processing procedures of the fourth embodiment are the same as those of the first to third embodiments, and a description thereof will be omitted.

It is needless to say that the fourth embodiment and the above-described embodiments can be used in an appropriate combination. By these combinations, for example, when the user is looking at a certain position, it is a nearby position where the user can check the content without greatly moving the line of sight, and the current user is using and referencing It is possible to present guide information at a position where it does not overlap with a display element on the screen or has a minimum overlap.

Further, subsequently, when the user moves his / her gaze to another display element near the display element currently referred to, for example, to confirm the contents of a different display element, It is also possible to control so that the input guide is moved to a position where it does not overlap or has a minimum overlap with the display element being referred to and where the user can confirm the content without greatly moving his or her eyes. is there.

It goes without saying that the fourth embodiment may be implemented in combination with the above embodiments as appropriate.

According to the fourth embodiment, the following effects can be further obtained in addition to the above embodiments.

The search state detection unit 408 monitors the search state of the user. The control unit 405 performs extension of the input waiting time, adjustment of the presentation time of the input guide information, and the like according to the search situation. Thereby, in a state where the user is examining or selecting the expression / content to be input while looking at the input guide, the input standby is appropriately extended. Therefore, an interface that is easier to use with less load is realized.

[0155]

As described above, according to the present invention,
The following effects are obtained. That is, the present invention provides a function of appropriately arranging guide information for assisting a user's input at a position determined based on gaze information indicating a position of a user's line of sight. In addition, a function of appropriately supplementing guide information with other guide information such as voice based on the user's gaze information is provided.

As a result, it is possible to efficiently and appropriately guide the input while reducing the burden on the user.

As described above, according to the present invention, in a multi-modal interface environment, the accuracy of data transfer in the dialog between the user and the computer is improved, and the natural and smooth communication between the user and the computer is achieved. Communication is realized.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a functional configuration of a multimodal interface device according to a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a hardware configuration of a computer system in which the multimodal interface device according to the present invention is mounted.

FIG. 3 is a flowchart illustrating a processing procedure performed by a control unit 105 in the multimodal interface device according to the first embodiment of the present invention.

FIG. 4 is a flowchart showing a procedure of a presentation correction process performed by a control unit 105 in the multimodal interface device according to the first embodiment of the present invention.

FIG. 5 is a diagram illustrating an example of an operation of the multi-modal interface device according to the user's line of sight according to the present invention.

FIG. 6 is a block diagram illustrating a functional configuration of a multimodal interface device according to a second embodiment of the present invention.

FIG. 7 is a block diagram illustrating a functional configuration of a multimodal interface device according to a third embodiment of the present invention.

FIG. 8 is a block diagram illustrating a functional configuration of a multimodal interface device according to a fourth embodiment of the present invention.

[Explanation of symbols]

 1, 2, 3, 4 Multimodal interface device 101 Output unit 102 Input unit 103 Visual guide presentation unit 104 Gaze target detection unit 105, 205, 305, 405 Control unit 206 Application management unit 307 Auxiliary voice presentation unit 408 Search state detection unit

Claims (11)

[Claims] 1. A multi-modal interface device for inputting / outputting information between a user and a computer in a plurality of interactive modes, wherein gaze information is generated as gaze information by detecting a gaze point of the user. Generating means; guide information control means for determining a position to present guide information for assisting the user's input based on the gaze information; and guide information control at the position determined by the guide information control means. A multi-modal interface device comprising: 2. The system according to claim 1, wherein the guide information control unit further includes: a control unit configured to determine whether a distance between a gazing point of the user indicated by the gazing information and a presentation position of the guide information is within a predetermined first threshold. Comprising a first determination unit for determining whether or not the guide information is to be set in the vicinity of the gazing point of the user when the distance is not within the range of the first threshold. The multi-modal interface device according to claim 1, wherein: 3. The guide information control means further comprising: fixing a presentation position of the guide information when it is determined that the user pays attention to the guide information. Item 3. The multimodal interface device according to item 1 or 2. 4. The guide information control means further comprises: a second determination means for determining whether the obtained presentation position of the guide information is within a predetermined presentation area where the guide information is to be presented. The multimodal interface device according to any one of claims 1 to 3, further comprising: when the presentation position is not in the presentation region, correcting the presentation position of the guide information in the presentation region. . 5. The guide information control unit according to claim 1, wherein the guide information control unit corrects a size of the guide information so as to present the guide information in a visual field of the user. Multimodal interface device. 6. The multi-modal interface device, further comprising: an application that generates application presentation information indicating at least one of a usage status of each application presented in a presentation area and arrangement information of each presentation element of the application. 6. The apparatus according to claim 1, further comprising a status grasping unit, wherein the guide information control unit sets the guide information to a position that does not hinder the presentation of each application, based on the application presentation information. 7. Or a multimodal interface device according to any one of the preceding claims. 7. The multi-modal interface device further comprises: auxiliary voice generation means for generating voice guide information based on a voice signal for assisting an input from a user; and output control for controlling output of the voice guide information. A guide to present the voice guide information when it is determined that the user cannot capture the guide information in the vicinity of the visual field based on the gaze information. 7. The multimodal interface device according to claim 1, wherein the information is set in information. 8. The multi-modal interface device further determines, based on the gaze information, whether or not the gaze point of the user stays in the area of the presented guide information. A determining unit, wherein the guide information control unit, when it is determined that the gaze point of the user is staying, the control unit maintains a standby state of an input from the user The multimodal interface device according to any one of claims 1 to 7, wherein: 9. The multimodal interface device according to claim 1, wherein said presentation control means controls the presentation of said guide information in a presentation area by translucent display or highlighted display. 10. A multimodal interface method for inputting and outputting information between a user and a computer in a plurality of interactive modes, comprising the steps of: detecting a gaze point of the user and generating the gaze information; A multi-modal method comprising: a step of obtaining a presentation position at which guide information for assisting the user's input is presented based on the gaze information; and a step of presenting guide information at the presentation position. Interface method. 11. A computer-readable recording medium storing a multi-modal interface program for inputting / outputting information between a user and a computer in a plurality of interactive modes, and detects a point of interest of the user. A module that generates a gaze information and a module that determines a presentation position for presenting guide information for assisting the user's input based on the gaze information; and a module that presents guide information at the presentation position. A computer-readable recording medium storing a multi-modal interface program, comprising: