JP3099753B2 - User interface device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a user interface device of an interactive system, and more particularly to a user interface device having a guidance function for a user.

[0002]

2. Description of the Related Art There have been many proposals for a user interface of an interactive system. Above all, a user interface that is easy to use even for those who are not accustomed to using a computer is a major problem, and there are various proposals using a pointing device and a voice input / output device.

[0003] However, many of these conventional proposals focus on work efficiency, easy-to-understand display contents, adaptability to user proficiency, and the like. It was a technique for smoothing. On the other hand, when the user encounters a problem or loses his understanding of what to do next, the system side hardly provides a means for assisting the user. Many existing systems can explain operations by referring to the help menu, etc., but do not have the function to support user's decision making or introduce the usage and services in appropriate situations .

[0004] Kamio et al., "Interaction apparatus and operation guidance output method applied to the apparatus" (Japanese Patent Application Laid-Open No. H08-044520).
Japanese Patent Application Laid-Open No. HEI 4-344930) and Hamada's "voice guidance output method" (Japanese Unexamined Patent Publication No. 4-344930), in order to provide appropriate guidance to the user, from the time when the user enters an operable state until the user performs some input. The user's proficiency is estimated by measuring time, and the content of the guidance is changed in accordance with the proficiency. However, if the user does not enter anything, it does not convey useful information according to the situation to the user. It does not necessarily provide information to support decision-making or problem solving.

Also, in Tsuchiya et al., "Emphatic Response Method in Information Retrieval" (Japanese Unexamined Patent Publication No. 2-201575), when a user fails to search the database, a response to prompt the user for the next input is presented. Facilitates dialogue. However, it only prompts for an input, and cannot arrange response contents according to the scene, and cannot provide further presentation when the first presentation content does not help the user to make a decision on the next input. In addition, no guidance is given if the user has been unable to do anything before the failure, which is not enough to completely eliminate anxiety for beginners.

[0006]

As described above, the user's question and anxiety about what kind of service can be provided and what to do next in the user interface device of the conventional interactive system. On the other hand, there is no means for presenting guidance according to the situation or presenting another content at an appropriate timing after one guidance is not sufficient.

Further, even if a plurality of guidance contents are prepared, simply presenting them one after another or simply presenting them at regular time intervals can be said to be guidance in consideration of the user's convenience and mental state. Absent.

Accordingly, it is an object of the present invention to provide a user interface device which solves the above-mentioned problem of the conventional user system of the interactive system.

[0009]

Means for Solving the Problems An input means for inputting by a user's operation and an output means for outputting information to be provided to the user are provided, and a dialog between the interactive application apparatus and the user is provided. In the supporting user interface device, a time measuring means for measuring a time elapsed after the user stops performing the input operation, a content of the guidance to the user, and an elapsed time as a condition for presenting the content to the user. And a guidance knowledge management unit that manages guidance knowledge including a pair of, when the time measured by the time measurement unit reaches one of the elapsed times included in the guidance knowledge, the elapsed time corresponds to the elapsed time. Automatic guidance means for providing the guidance to the user.

[0010] Further, there is provided state management means for managing the internal state of the application apparatus, wherein the auto-guidance means detects the internal state of the application apparatus from the state management means, and receives the internal state from the guidance knowledge management means. And provide guidance to the user by selecting the guidance knowledge according to.

Further, there is provided application information retrieval means for retrieving information corresponding to each internal state of the application apparatus, wherein the automatic guidance means obtains the contents of the guidance obtained from the guidance knowledge from the application information retrieval means. Information to the user.

Further, anthropomorphic response means for outputting the response content to the user by voice and gesture, text output means for displaying the response content of the natural language sentence to the user, and voice output to the user Response knowledge storage means for storing response knowledge describing each response content for a text output, a gesture operation, the auto-guidance means,
Guidance is performed using the anthropomorphic response means and the text output means.

Further, the guidance knowledge includes an item indicating whether to continue the measurement of the time by the time measuring unit or return to the initial value after executing the guidance according to the content of each guidance, and According to the item in the guidance knowledge, instructs the time measurement means to continue the time measurement, or instructs the elapsed time to return to 0, and further, the time measurement means, the instruction from the auto guidance Alternatively, when it is detected that there has been any input from the user, the elapsed time is returned to 0 and time measurement is started again.

Further, an instruction input means for indicating a position on the display screen or an item selected by the user,
Character string input means for a user to input a character string; menu selection means for displaying a list of items that can be instructed to the user interface device and the application device; 2
Dialogue control means having at least one dialogue mode, and simultaneously controlling the dialogue in any one of the dialogue modes;
Interactive mode switching means for switching to an interactive mode desired by a user among the interactive modes in an arbitrary scene by the character string input means.

The apparatus further comprises natural language analyzing means for analyzing an input character string described in a natural language and sending a result to the interactive control means. The input character string from the character string input means and the menu selection means Are analyzed together with the selection result character string by the natural language analysis means.

Further, the natural language analysis means specifies the object from the internal state of the application managed by the state management means when the object of the operation or question is not specified in the input content from the user. .

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION In the following, an embodiment of the present invention will be described in which a microphone, a keyboard, a touch panel, and a mouse are used as input means, a display device and a speaker are used as output means, and a library information search system is an application device. This will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of a user interface device according to an embodiment of the present invention and an application device to which the user interface device is applied.

The user interface device shown in FIG.
Keyboard 1, voice recognition device 2, touch panel 3, mouse 4, display device 5, speaker 6, character string input unit 8, instruction input unit 9, natural language analysis unit 10, menu selection unit 11, dialog control unit 12, state management A unit 13, an auto-guidance unit 14, a guidance knowledge management unit 15, a time measurement unit 16, an application information search unit 17, a text output unit 19, and an anthropomorphic response unit 20. Connect and work.

The voice recognition device 2 is a device that recognizes a user's voice input and converts it into a corresponding character string. The voice recognition function may be realized by a device dedicated to voice recognition connected to a microphone, or may be implemented by software in the same device as the user interface device 2.

The touch panel 3 enables a user to specify an arbitrary position on the display device 5 with his / her finger or the like, thereby pressing a button displayed on the display device 5, selecting a menu, specifying an area, and the like. Device.

As shown in FIG. 2, for example, the display device 5 outputs the visual information output by the application device 7 (211), the output from the text output unit 19 (212), and the output from the anthropomorphic response unit 20. This device is used for displaying visual information output contents (213), menus (214) by the menu selection unit 11, and other visual information presented to the user as needed.

The speaker 6 is a device that outputs audio information from the anthropomorphic response unit 20 and other auditory information presented to the user as necessary.

The application device 7 is a book information retrieval system in the present embodiment, and has a function of extracting necessary information from the application database 18 and presenting it to the user according to a user's instruction. Have. Further, the application device 7 has a plurality of states that can be transited to each other by a user operation, and can output each state as a variable value to an external device.

The character string input unit 8 converts a character string input from the keyboard 1 or the speech recognition device 2 into a natural language analysis unit 1.
Pass to 0.

The instruction input unit 9 includes a menu selection unit 11 out of information input from the touch panel 3 or the mouse 4.
Informs the menu selection unit 11 that the user has selected one of the menus displayed on the display device 5.

The menu selection unit 11 displays on the display device 5 a menu corresponding to the interaction mode determined by the interaction control unit, and displays the contents of the menu selected by the user via the instruction input unit 9 in text. Natural language analysis unit 10 as a column
Send to

The natural language analysis unit 10 analyzes the character string sent from the character string input unit 8 or the menu selection unit 11 as a natural language sentence or word, and converts it into an analysis result that can be processed by the dialogue control unit 12. It is sent to the dialogue control unit 12.

The dialogue control unit 12 acquires the state of the application device 7 via the state management unit 13 if necessary according to the analysis result sent from the natural language analysis unit or the instruction of the auto guidance unit, or retrieves the application information. Necessary data is acquired from the application database via the unit 17 and an operation instruction is sent to the application device 7 or a response is made to the user using the text output unit 19 and the anthropomorphic response unit 20.

The state management unit 13 acquires the state information of the application device 7 in accordance with a request from the natural language analysis unit 10 or the dialog control unit 12 and sends it to the request source.

The auto-guidance unit 14 includes the time measurement unit 1
6 detects the elapse of the time designated in advance and notifies the dialog control unit 12, and extracts the guidance knowledge corresponding to the elapsed time from the guidance knowledge management unit 15. An appropriate response content is selected from the extracted guidance knowledge, and guidance is provided to the user using the text output unit 19 and the anthropomorphic response unit 20.

The time measuring section 16 is provided for the automatic guidance section 1.
According to the instruction from 4, the measurement of the time is continued after the guidance or the elapsed time is returned to 0. When the user performs any input operation, the elapsed time is immediately returned to 0, and the time measurement is started again.

The application information retrieval unit 17 sends the dialogue control unit 1 to the application database 18 which manages information used by the application device 7.
The search is performed in response to a request from 2. The application database 18 stores, for example, bibliographic data of each book, spatial position information indicating which book is placed in which room in the library, classification and characteristics of each room and book shelf, and the like. .

The text output unit 19 displays on the display device 5 a response sentence generated based on the guidance knowledge and the information obtained from the application database.

The anthropomorphic response unit 20 is described in, for example, Lu's "User Interface Method and Apparatus by Displaying a Person Image" previously filed by the present applicant (Japanese Patent Application No. 7-232638).
Using an anthropomorphic character as described in the above, a message is transmitted to the user by voice and gesture.

Next, a specific operation of the present invention will be described.
This will be described in more detail with reference to the drawings.

First, a more specific example of the state management unit 13 will be described with reference to FIG.

FIG. 5 shows a portion of the user interface device according to the present embodiment when a user reading a book displayed on a display screen as an example of an input sentence inputs "What is the shelf on which this is placed?" Shows a typical processing example. In the example of the input sentence, the part indicating the target of "placed" is represented by the pronoun "this", and the target is unclear. In this case, the natural language analysis unit 10 refers to the value of the function parameter representing the function of the word “this” when the analysis dictionary 412 is referred to,
The value ind (short for indicate) is obtained. fu
When the value of the nction parameter is ind, the natural language analysis unit 10 requests the state management unit 13 of the objects output by the application device 7 that can be the target indicated by the “this”. Upon receiving the request, the status management unit 13 requests the latest status information from the application device 7 and stores the obtained status information in the status management table 413. According to the status information stored in the status table 413, the natural language analysis unit 10 receives the highest priority value among the values in the table that are not NULL according to the priority order 42, and the semantic frame 411 corresponding to the “this”. Is stored as the value of the Value parameter. Here, the meaning frame is a table that stores information unique to each word constituting the input character string. The “>” symbol in the priority order 42 is a symbol for convenience indicating that the left side has priority over the right side.

Next, the operation of the auto guidance unit 14 and the time measurement unit 16 will be described with reference to FIGS.
This will be described more specifically with reference to FIG. As shown in the flowchart 51 of FIG. 6, when the user interface device according to the present embodiment is activated, the time measurement unit 16 first reads the guidance start set time using each guidance knowledge from the guidance knowledge management unit 15. (S11). Here, the guidance start set time is, for example, in the case of the guidance knowledge 31 in FIG. 3, the elapsed time (waittime) is set to 1
0 (seconds) is described (311), which is extracted and read into the time measuring unit 16. Guidance knowledge management unit 15
When the set time is read for all the guidance knowledge managed in the above, the elapsed time is initialized (the elapsed time is set to 0).
(S12), and time measurement is started (S1).
3). Thereafter, it is determined whether or not there has been any input from the user (S14). If the input has been made, the elapsed time is initialized and the time measurement is restarted. If there is no input, it is determined whether the elapsed time has reached any of the set times (S15). When the elapsed time has reached any of the set times, the elapsed time is passed to the auto guidance unit, and the automatic guidance is started (S16). If the elapsed time has not reached any of the set times, the time measurement is continued. When the auto-guidance is completed, it is determined whether to continue or initialize the time measurement based on the guidance knowledge applied in the auto-guidance, and each is executed (S
17). Regarding the determination of whether to continue or initialize the time measurement, for example, as described in the guidance knowledge 31 and the guidance knowledge 33 in FIG. 3, the attribute value of the elapsed time (timer) is set to continue (31).
If it is 3), continue, and if reset (331), it is determined by referring to the guidance knowledge, such as initialization.

As shown in a flowchart 52 of FIG. 7, the auto guidance unit 14 reads one or more pieces of guidance knowledge corresponding to the elapsed time received from the time measurement unit from the guidance knowledge management unit 15 (S21), and sets each guidance. Determine if the knowledge has an applicable condition. For example, in the guidance knowledge 31 of FIG. 3, the application condition is that the attribute value of the state information (focus) is 0001013R (3
14). As for the guidance knowledge having the application conditions, corresponding state information is acquired from the application 7 via the state management unit 13 (S23 in FIG. 7). Then, the state information is compared with the application conditions, and only the guidance knowledge that matches them is selected and sent to the next process (S24). Guidance knowledge with no applicable conditions is sent to the next process as it is. Further, it is determined whether or not the guidance knowledge sent to the next process includes a variable requiring application information (abbreviated as AP information in FIG. 7) in the guidance output (S25). Application database 18 using information retrieval unit 17
From the application information required (S2)
6). As an example of the application information, the floc included in the output content 315 of the guidance knowledge 31 in FIG.
Fgenre (classification name of the corresponding floor) (32) included in the output contents of (the number of floors of the corresponding floor) and guidance knowledge 32
1) and so on. After retrieving these application information as necessary and substituting the variables, the output contents in the corresponding guidance knowledge are sent to the text output unit 19 and the anthropomorphic response unit 20 to execute the guidance (S
27). For example, when the guidance knowledge 31 of FIG. 3 is applied, if the search result of the floc is “third floor”, the text output unit 19 sends the sentence “This is the third floor” (31).
5), but the operation number (action
The gesture and audio output (312) defined by the attribute value 108 of n) are sent.

The relationship between the execution of each guidance and the lapse of time is as follows. When the guidance knowledge 31, 32, and 33 in FIG. 3 are continuously executed with elapsed times of 10, 30, and 300 seconds, respectively, FIG. It looks like 34. After the guidance 1 (guidance based on the guidance knowledge 31) and the guidance 2 (guidance based on the guidance 32) are respectively performed, if a total of 300 seconds elapse without any input from the user, the guidance knowledge 33 is applied and the operation number (action) )
The state of the user interface device and the state of the application device 7 of this embodiment are initialized by the attribute value 130 of this embodiment, the elapsed time is initialized to 0, and the time measurement is restarted.

In the dialogue with the user by the user interface device of the present embodiment, it is assumed that there are two or more modes of the dialogue. The switching of the interactive mode can be visually confirmed by the difference in the menu display contents by the menu selection unit 11 in FIG. Hereinafter, each interactive mode is represented by the corresponding menu display contents.

FIG. 8 is an example in which the transition of the menu display contents by switching the interactive mode is indicated by an arrow.
The user can switch each of the interactive modes 61, 62, 63, and 64 by using the touch panel 3 and the mouse 4 in accordance with solid arrows. The user can select a menu displayed in the interactive mode 61 (“guidance inside the building” or “book search”).
, Etc., by using the touch panel 3 or the mouse 4 to shift to another interactive mode. In each of the interactive modes 62, 63, 64, selecting the “completion” menu returns to the interactive mode 61. In this case, be sure to use interactive mode 6
The user cannot go to another interactive mode without going through 1.

Here, in the user interface device of the present embodiment, a menu item such as "guidance in the hall" in the interactive mode 61 is input as a character string by the keyboard 1 or the voice recognition device 2, whereby the character string is changed. It is input to the natural language analysis unit 10 via the character string input unit 8.
As a result, the character string is analyzed in the same manner as when the input from the menu selection unit 11 through the touch panel 3 or the mouse 4 is input to the natural language analysis unit 10, and in any of the interactive modes, a transition is made to another interactive mode. (Represented by dashed arrows). Further, the user is not limited to the interactive mode 61, and can input a menu item name such as “Register a book” or a request sentence in a natural language such as “I want to register this book as a personal book” as a character string. Interactive mode 6 in any interactive mode
4 and a specific operation can be started.

[0045]

By applying the user interface device of the present invention, even if the user of the interactive system falls into a state where it is difficult to determine what to do next, guidance according to the situation at that time is provided. Can be received. Further, in the present user interface device, even if the content of the presented guidance is not sufficient for the user's decision making, the user's decision making is continuously performed by performing another guidance at an appropriate time. Can help. Furthermore, even in applications with multiple interactive modes,
With the mouse and touch panel, only the menus and buttons limited to each interactive mode can be operated, saving the screen display area and preventing erroneous operations, and using the voice and keyboard, the flexibility to directly shift to any interactive mode is also available. Prepare.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a user interface device according to an embodiment of the present invention and an application device to which the user interface device is applied;

FIG. 2 is a diagram showing an example of screen output according to the embodiment;

FIG. 3 is a diagram illustrating guidance knowledge according to the embodiment;

FIG. 4 is a diagram showing the relationship between the execution of guidance and the passage of time according to the embodiment;

FIG. 5 is a diagram illustrating a partial processing example of the user interface device according to the present embodiment;

FIG. 6 is a flowchart showing the operation of the time measuring unit of the present invention.

FIG. 7 is a flowchart showing the operation of the auto guidance unit of the present invention.

FIG. 8 is an example in which transitions of menu display contents are indicated by arrows.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Keyboard 2 Voice recognition device 3 Touch panel 4 Mouse 5 Display device 6 Speaker 7 Application device 8 Character string input unit 9 Instruction input unit 10 Natural language analysis unit 11 Menu selection unit 12 Dialogue control unit 13 State management unit 14 Auto guidance unit 15 Guidance Knowledge management unit 16 Time measurement unit 17 Application information search unit 18 Application database 19 Text output unit 20 Anthropomorphic response unit 21 Screen output example 211 Screen output example of book information retrieval system 212 Text output example 213 Anthropomorphic response example 214 Menu output Example 31 Guidance knowledge 32 Guidance knowledge 33 Guidance knowledge 34 Relationship between execution of each guidance and passage of time 311 Elapsed time designation attribute and attribute value 312 Operation number attribute and attribute value 313 Elapsed time attribute And attribute value 314 Status information attribute and attribute value 315 Text output attribute and attribute value 321 Text output attribute and attribute value 331 Elapsed time attribute and attribute value 42 Priority of status information 411 Meaning frame 412 Analysis dictionary 413 Status management table 51 Time measurement Section Flowchart 52 Oh and Guidance Section Flowchart 61 Interactive Mode 62 Interactive Mode 63 Interactive Mode 64 Interactive Mode

──────────────────────────────────────────────────続 き Continuation of front page (56) References JP-A-5-53705 (JP, A) JP-A-8-16348 (JP, A) JP-A-6-214741 (JP, A) JP-A-6-14741 110634 (JP, A) JP-A-4-370825 (JP, A) JP-A-4-354013 (JP, A) JP-A-1-284915 (JP, A) (58) Fields investigated (Int. ⁷ , DB name) G06F 3/14 330-340 G06F 3/00 653-654 G06F 3/02 370

Claims (7)

(57) [Claims] An input means for inputting by a user operation
And means for outputting information to be provided to the user.
Prepared for the interaction between the interactive application device and the user.
In the supporting user interface device, the time elapsed since the user stopped performing input operation
Time measurement means to be measured and contents of guidance to usersAnd thatContent to users
Elapsed time as a condition to be presentedCorrespondedguidance
Guidance knowledge management means for managing knowledge, and time measured by the time measurement meansBeforeChronicle Guida
If one of the elapsed times included in theTo
PreviousProvide the user with the guidance corresponding to the elapsed time
Auto guidance means,Of the application device
State management means for managing the internal state, The auto-guidance unit is located before the state management unit.
The internal state of the application device is detected and the
Select guidance knowledge according to the status and provide guidance to the user
Including means for performing The guidance knowledge follows the content of each guidance
After execution of the guidance, the time is measured by the time measuring means.
An item indicating whether to continue measurement or return to the default
Each including The auto-guidance means includes:
According to the attribute value of the item,
To continue time measurement or reset elapsed time to 0
Including means to User interface characterized by the following:
apparatus. 2. The application device of claim 1,Each internal state
Application information searcher to search for information corresponding to
With steps, The auto-guidance means uses the guidance knowledge
Search the application information based on the content of the obtained guidance
Means to add information obtained from the means and present it to the user.
M Claim1 noteUser interface device. (3)Voice and gestures to respond to the user
Anthropomorphic response means for outputting Text output showing response content of natural language sentence to user
Power means, For voice output, text output, gesture
Response knowledge that describes the contents of each response for
Answer knowledge storage means, With The auto-guidance means is provided in front of the personified response means.
Including means for providing guidance using text output means.
M Claims characterized by the following:2 notesUser interface
Equipment. 4. The time measuring means includes an automatic guidance
The above instructions from the user or any input from the user
When it is detected that the elapsed time has been
Beauty user interface device according to claim 3 Symbol mounting includes means for initiating the time measurement. Claim 5.The position on the display screen or the user
Instruction input means for indicating the selected item; Character string input means for a user to input a character string; The user interface device and the application
Menu that displays a list of items that can be
Selection means, Two or more for various purposes in the dialogue with the user
It has an interactive mode and at the same time, any one interactive mode
Dialog control means for controlling the dialog by limiting; The character
Use the interactive mode at any time by using the column input
Interactive mode that switches to the interactive mode desired by the user
Switching means; With Claim4 notesUser interface device. 6.Parse input strings written in natural language
And a natural language analysis means for sending a result to the dialog control means.
Prepare, This natural language analyzing means receives input from the character string input means.
Input character string and the selection result sentence from the menu selection means
Includes means to analyze strings together Claim5 notesListed users
Interface device. 7. The natural language analysis means according to claim 1, wherein
The target of the operation or question is clearly specified in the input contents.
If not, the application managed by the status management means
® down user interface device of claim 6 Symbol mounting includes means for identifying the object from the internal state of.