JP2000112610A - Contents display selecting system and contents recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a contents display selecting system by which the intension and command of a user who operates a computer are received through the use of plural recognizing methods including not only voice recognition but also at least picture recognition and also linking is executed to a designated link destination. SOLUTION: In this system, a 'category' indicating the kinds of recognizing devices, 'recognition candidate data' as the selection matter of the user, which is to be recognized by the recognizing devices, and 'link information' being the link destination of a selection result are described in the description text of the contents, a control means 10 transmits the 'category' provided in display contents displayed in a display device 12 to the recognizing devices 14A and 14B, 'recognition candidate data' are transmitted to the recognizing devices so as to be recognized, applying link information is selected by receiving the recognition result and the display contents are changed into the contents indicated by link information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to display of data by a computer and reception of commands and the like from a computer user. The present invention relates to a recorded content recording medium.

[0002]

2. Description of the Related Art Computers have become compatible with multimedia, have been used in networks via communication lines and the like, and with the spread of the Internet,
Computer users are increasing exponentially. Heretofore, commands have been given to computers mainly using manually operated instruments such as keyboards and mice. A system has been developed that recognizes voices generated by humans in place of such manually operated instruments and accepts wills and commands of computer users. A speech interactive computer system using such speech recognition is disclosed in, for example, Japanese Patent Application Laid-Open No. 8-335160. According to the prior art described in this publication, a means for recognizing a video screen display signal and a data link set obtained by a data connection is provided, a grammar corresponding to the data link is obtained, and the grammar is converted to a voice recognition unit. It can be received at.

[0003] As another conventional technique, IE95-46, MVE95-39 (1995, July), IEICE's IEICE-Technology, extracts information necessary for mechanical recognition from existing content description languages. A technique for performing this is disclosed.

[0004]

In these prior arts, only a limited recognition method can be selected in a limited system. That is, Japanese Patent Application Laid-Open No. 8-335
In the method of Japanese Patent Publication No. 160, it is possible to accept a user's intention or command only by voice recognition. For example, by performing recognition of gesture, sign language, facial expression, handwritten pattern, etc., the same recognition can be simultaneously performed for a plurality of recognition means. No description has been made of a technique of describing the content description text on the content description text and accepting the user's intention or command using the plurality of recognition means.

[0005] Further, in the mechanical extraction, as described on page 36 of the above-mentioned IEICE technique, if another link destination is defined by the same character string, which content (display part) is I do not know if I can specify it by voice,
There is a problem. Even if special recognition is possible, it is difficult to understand the operation method without familiarity with the system or without special training, and there is also a problem that it is difficult for a beginner to handle. In addition, from the viewpoint of assisting handicap users who are handicapped, it is desired to use a plurality of recognition methods. For an Internet business provider that provides a content browsing system such as a homepage, improvement of the content itself is required. In response, there is a demand for improvement of the hardware side of the user who browses the content.

Accordingly, the present invention can accept a user's intention or command for operating a computer using at least a plurality of recognition methods including image recognition as well as voice recognition, and can specify a link destination specified according to a user's instruction. It is a first object of the present invention to provide a content display selection system that can link to a content display selection system.

In addition, the present invention can accept a user's intention or command for operating a computer using at least a plurality of recognition methods including image recognition as well as voice recognition, and can specify a link destination specified according to a user's instruction. It is a second object of the present invention to provide a content recording medium on which content for enabling linking to a content is recorded. In the present invention, “image recognition” is a concept including recognition of gestures, sign language, facial expressions, handwritten patterns, and the like of a user who operates a computer. Therefore, it includes not only optical recognition using a video camera or the like but also recognition by any image recognition means, such as shape recognition using pen pressure or the like.

[0008]

In order to achieve the above object, according to the present invention, contents to be displayed on a display device of a computer recognize a user's will by one of a plurality of recognition methods including at least voice recognition and image recognition. In order to be able to do so, a description for giving the category of the recognition method by voice or image is included in advance, and when this content is displayed by the content display selection system as a computer on which the browser software is installed, at least this computer A configuration including a recognition device including voice recognition and image recognition, a display device for displaying the content, and control means for controlling the recognition device and the display device in association with each other. , "Category" indicating the type of recognizer,
"Recognition candidate data" which is a candidate for a user selection item to be recognized by the recognition device and "link information" which is a link destination of the selected item are described as recognition information, and the control means is displayed on the display device. Sending the "recognition candidate data" to the recognition device so that the "recognition candidate data" displayed on the display device can be recognized by the recognition device; In response to the result of the recognition, the corresponding link information is selected, and the display device is controlled to change the content displayed on the display device to the content indicated by the corresponding link information.

That is, according to the present invention, a content display selection device including at least a recognition device including voice recognition and image recognition, a display device for displaying content, and control means for controlling the recognition device and the display device in association with each other. In the system, the description text describing the content to be displayed includes “category” indicating the type of the recognition device.
And "recognition candidate data", which is a candidate for a user selection item to be recognized by the recognition device, and "link information", which is a link destination of the selected item, are described as recognition information. The "category" of the display content displayed on the display device is sent to the recognition device, and the "recognition candidate data" displayed on the display device is displayed.
To be recognized by the recognition device, send the "recognition candidate data" to the recognition device, and receive the result of recognition by the recognition device, select the corresponding link information, and display the content displayed on the display device. There is provided a content display selection system configured to control the display device to change to the content indicated by the corresponding link information.

Further, according to the present invention, a description for giving a category of a recognition method by voice or image so that a user's will can be recognized by at least one of a plurality of recognition methods including voice recognition and image recognition. A content recording medium is provided in which the content including the content is recorded in a computer-readable state.

[0011] In the description text, "auxiliary information" for displaying a supplementary character string relating to the recognition candidate data is described as the recognition information, and the control means displays the "auxiliary information" on the display. It is a preferred aspect of the present invention that the display device is configured to send a supplementary character string relating to the recognition candidate data on a screen by the "auxiliary information" by the "auxiliary information".

In the description text, "guide information" indicating a recognition method included in the recognition device is described as the recognition information, and the control means sends the "guide information" to the display device, It is a preferred aspect of the present invention that the display device is configured to show a recognition method included in the recognition device on a screen by the "guide information" and provide a guide of the recognition method to a user.

Further, it is a preferred embodiment of the present invention that the image recognition is for recognizing at least one of a gesture, a sign language, a facial expression, and a handwritten pattern.

According to the present invention, when displaying a homepage such as the Internet, in addition to specifying the displayed content with a keyboard, a mouse, or the like, a voice check,
The description text of the homepage includes a recognition system category and a format that describes information on related content so that selections can be made using gestures, sign language, facial expressions, handwritten patterns, etc. The information necessary for recognition is acquired from the description text according to the information.

Further, for each corresponding recognition system, guide data indicating what kind of voice recognition, gesture, sign language, facial expression, and handwritten pattern recognition are provided when displaying a home page is provided. It is a preferred embodiment of the present invention to simultaneously provide, for example, an announcement by voice, a gesture, a sign language, an animation for facial expression, a display by moving image data, and a handwritten pattern to display a reference graphic (including animation).

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram conceptually showing a preferred embodiment of a content display selection system according to the present invention. This content display selection system is composed of a computer and its related devices. When this is functionally shown, a CPU (Central Processing Unit) not shown, a control device 10 including a main storage device and an interface, and an image display device Display device 12 for reproducing voice, first recognition device 14A for performing voice recognition, second recognition device 14B for performing image recognition, link information storage section 16 for storing link information, auxiliary information display for controlling display of auxiliary information The state where the control unit 18 is present and receives the content description text 20 from the server 22 is shown.

First, the HTM is stored in the main memory of the control device 10.
It is assumed that browser software for displaying the L-description content is installed in advance. The second recognition device 14B recognizes one or more of gestures, sign language, handwritten patterns, and the like by the upper body or the whole body of the user. Therefore, by using the first recognition device 14A and the second recognition device 14B together, voice recognition and gesture recognition, voice recognition and sign language recognition, voice recognition and handwritten pattern recognition, or voice recognition and gesture recognition, sign language recognition, handwritten pattern The user's will or command can be recognized by two or more of the recognitions.

FIG. 2 is a flowchart showing the operation of the configuration of FIG. 1. The operation of the configuration of FIG. 1 will be described with reference to this flowchart. The browser software starts up,
It is assumed that the program of FIG. 2 has started, and the control device 10 has acquired the content description text 20 from the server 22 via a network, a file system, and the like (step S1). The control device 10 analyzes the content description text 20 (step S2) and determines whether or not the extended recognition information is included (step S2).
3). If the extended recognition information is not included, that is, if the content is a normal content, an image, a text, and the like are displayed by the conventional display processing (step S).
4).

On the other hand, when the extended recognition information is confirmed,
A search is performed to determine whether recognition information that matches the recognition method of the recognition devices 14A and 14B connected to the control device 10 is included, and the recognition method of the prepared recognition devices 14A and 14B (speech recognition and image recognition in the example of FIG. 1). When the recognition information corresponding to ()) is found, the control device 10 extracts the recognition information from the content description text (step S5). Next, recognition candidate data corresponding to the first recognition device 14A and the second recognition device 14B is sent (steps S6A and S6B). At the same time, the link information storage unit 16 stores the link destination related to the recognition candidate. In addition, these steps S6A, S6B
Then, the corresponding “auxiliary information” described in the content description text is sent to the auxiliary information display control unit 18. Next, the control device 10 issues a recognition start command to the recognition device 14.
A and 14B (step S7). In response, the recognition devices 14A and 14B start recognition processing based on the recognition information sent from the control device 10.

Next, it is checked whether or not recognition has been performed by each of the recognition devices 14A and 14B (step S).
8). In this flowchart, the two recognizing devices 14A and 14B are checked in order. However, when there are many recognizing devices or when a long time is required for checking, an interrupt process can be used. When it is determined in step S8 that the recognition has been performed, the recognition devices 14A and 14B return to the control device 10 which of the corresponding recognition candidates has been recognized, and the control device 10 returns the recognition result. Is displayed on the display device 12 (step S9),
The control device 10 determines the link information storage unit 16 from the recognition result.
Is searched (step S10), the corresponding link is selected, and a new content description text 20 is acquired from the server 22 as a new link (step S11). Note that the auxiliary information display control unit 18 uses the “auxiliary information” described in the content description text.
In response, the corresponding display is performed on the display device 12. The display device 12 reproduces not only images but also audio, and specifically includes a display and a speaker (headphone).

Next, a second embodiment of the present invention will be described with reference to FIGS. The second embodiment is effective when the content is described with “guide information” added thereto. That is, when “guide information” is added to the content, another data and content indicated by the guide information are acquired, and the guide information display control unit 24 is controlled based on the guide information such as audio, video, and animation.
Can control and reproduce the display device 12. That is, as shown in FIG. 3, in the second embodiment,
A guide information display control unit 24 is connected to the control device in addition to the configuration of FIG. This guide information display control unit 24
, The guide information 1 and the guide information 2 included in the content are temporarily stored, and the display mode on the display device 12 is controlled. Further, the flowchart of FIG. 4 is provided with steps S12A and S12B for sequentially displaying and reproducing the guide information 1 and the guide information 2 between steps S6B and S7 of the flowchart of FIG.

Next, an example of recognition information added to content will be described. The recognition information is information represented by “category”, “recognition candidate data”, “link information”, and “auxiliary information” added to the description text describing the content. The category describes the type of recognition device. Examples include voice recognition, gestures, sign language, facial expressions, handwritten patterns, and the like.

The recognition information for speech recognition provides speech candidates necessary for speech recognition. The control device 10 selects link information from the result recognized by the recognition device 14.
The gesture captures the motion of the upper body or the whole body of the user with a camera image, analyzes a series of the motion by motion capture or the like, and associates a link with the content on the screen. For gestures, a series of actions are standardized, such as "bow", "right hand up", "left hand up", "sit", "jump", "right hand waist", "left hand shake", "right hand up" If the operation such as the above is determined and the result of the operation analysis by the second recognition device 14B is standardized, there is further versatility.

For sign language, a series of actions has already been defined, and a simple action to a complex action can be selected by the analysis capability of the second recognition device 14B. In particular, when not recognized as sign language, it can be one of recognition by gesture. Regarding facial expressions,
Recognition is possible by a camera image or the like. As with gestures, there are "close right eye", "close left eye", "open mouth", "put out tongue" and so on. Handwritten patterns can be used as substitutes for keyboards by applying techniques such as handwritten character recognition. However, even if complete character recognition is not achieved, it is possible to use "maru", "x", "wavy",
Simple figures such as "Yama", "Square" and "Triangle" are sufficient.

The types of recognition devices corresponding to these various recognition methods are defined as categories, and each piece of recognition information is defined as follows. <Extend Recognition Category = ”xxxx”> xxxx: Category <Language = yyyyy> yyyy: Information on description language <List aaaa, bbbb, cccc> aaaa: Recognition candidate bbbb: Link destination cccc: Auxiliary information </ Extend Recognition>

The categories are defined as follows: "Voice" Voice recognition "Gesture" Gesture "Sign language" Sign language "Face" Facial expression "Pen" Handwriting pattern, tablet, etc. It is.

Since the control unit 10 ignores the extension that is not prepared for itself, if it is described in accordance with the above format, it is ignored even if there is a description about an unknown recognition device in the content description text.

The recognition information includes the following examples. In the case of speech recognition <Language = "Japanese"> The language system is Japanese <List "Rinkuichi", http://www.server1.com, "Rinku" 1 "> Reading candidate is" Rinkuichi "Link destination is http://www.server1.com When displayed on the screen, use the character string" Link1 "

In the case of the List shown here, when the voice "Rinkuichi" is recognized, the link destination http: //www.s
Display the contents of erver1.com. When a recognition candidate is displayed on the screen, a third character string "Rinku 1" is used. The language system indicates what language is used for notation of recognition candidates, and when providing candidates to recognition systems other than Japanese, several Language tags are prepared to enable support for other languages.

In the case of a gesture, <Language = "Japanese"> The operation description is Japanese <List "Right hand up", http://www.server1.com, "Right hand up"> The candidate for reading is "Right hand" Use the character string "raise right" to display the link destination on the http://www.server1.com screen.

Here, a Language tag is prepared in consideration of using a language other than Japanese for the description of the operation. Li given here
In the case of st, when the operation of raising the right hand is recognized, the content of http://www.server1.com which is the link destination is displayed. When displaying the operation candidates on the screen, a third character string “raise right hand” is displayed.

In the case of sign language, <Language = “Japanese”> Sign language system is Japanese <List ”Hello”, http://www.server1.com, “Hello”> Sign language candidate is “Hello” Link to http : //www.server1.com When displaying on the screen, use the character string "Hello"

The reason why a language is specified as a sign language system in sign language is that the definition of sign language differs from country to country.
A Language tag is also prepared here as a sign language system. In the case of the List given here, if the sign language action expressing "Hello" is recognized, the link destination is htt
Display the contents of p: //www.server1.com. When a candidate for the sign language action is displayed on the screen, a third character string "Hello" is displayed.

In the case of a facial expression, <Language = "Japanese"> The operation notation is Japanese <List "Open mouth", http://www.server1.com, "Open mouth"> The operation candidate is "Mouth open". Use the character string "Open your mouth" to display the link on the http://www.server1.com screen.

Here, too, a Language tag is prepared in consideration of using a language other than Japanese for the description of the operation. Li given here
In the case of st, when the operation of "opening the mouth" is recognized, the content of the link http://www.server1.com is displayed. When displaying facial expression candidates on the screen, a third character string “open mouth” is displayed.

In the case of a handwritten pattern, <Language = "Japanese"> is written in Japanese <List "maru", http://www.server1.com, "writes a circle"> The candidate for operation is "maru" Write the link http://www.server1.com When displaying on the screen, use the character string "Write a circle"

Here, too, a Language tag is prepared in consideration of using a language other than Japanese for the description of the operation. Other than Japanese, there are also English notations such as writing symbols directly and Draw Circle. In the case of List given here, if the action of writing a circle is recognized, the link destination is http: // w
Display the contents of ww.server1.com. Work on screen,
When displaying candidate symbols, a third character string "write circle" is displayed.

As described above, for various recognition methods,
A content description text is prepared so as to provide a recognition candidate, a link destination, and auxiliary information. When it is confirmed that the read content description text includes a candidate for the above recognition system, the control device 10 selects recognition information that matches the recognition system provided in its own control device 10 and selects candidate data. From the recognition unit of the recognition system. In the recognition system, when a registered operation or candidate is selected, which candidate is recognized is returned, and the control device 10 selects a link destination from the recognized result and displays the content of the link destination.

In the case of the second embodiment,
In the fourth parameter of the List notation, guide information for guiding the recognition candidate is described. For example, Li for voice recognition
If it is st, describe a waveform file that is audio data in the fourth parameter, such as <List ”Rinkuichi”, http://www.server1.com, “Link1”, Link1.wav> . In the display system, when the fourth parameter is confirmed, when displaying the content, the waveform file is reproduced by voice at the same time.

For example, if a page is displayed and a link can be designated by voice "Rinkuichi" by voice recognition, the content displayed on this page can be selected by voice "Rinkuichi". Audio guides can be added. When this includes a plurality of Lists, the control device 10 prepares a voice saying "The content displayed on this page can be selected by voice" and "Rink" from each List. Extract voices for candidates "ichi" and "rinkuni"
A guide is given by linking messages as follows: "The content displayed on this page can be selected with the voice of Rinkuichi."

In the case of gestures, facial expressions, and sign language, the fourth parameter describes data such as moving images and animations. <List “Right up”, http://www.server1.com, “Right Use it as video data guide information with audio, such as “raise”, motion1.avi>, and at the same time as displaying the content, “You can select the content displayed on this page by raising your right hand so that you can see it on the screen. "And the video can provide guidance for candidates. When a plurality of selections are possible, a plurality of candidates are guided by linking voice guidance as in the case of voice recognition.

Similarly, for sign language and handwritten patterns,
By displaying voice and moving image, animation, or still image candidates on the screen, even a novice user can operate without hesitation. In addition, depending on the control device 10, a plurality of pieces of such recognition information can be simultaneously handled,
"Speech recognition" with guidance system for guidance
It is possible to provide a plurality of “gesture” guidances.

[0043]

According to the present invention as described above, recognition candidate data and link information for a plurality of recognition devices can be provided on the same content description text. It is possible to obtain necessary recognition candidate data and link information according to the recognition device provided in the device. Also, by simultaneously providing and displaying the guide information, the user can be guided by the recognition means.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of a first embodiment of a content display selection system according to the present invention.

FIG. 2 is a flowchart showing an operation in the first embodiment of FIG.

FIG. 3 is a schematic block diagram of a second embodiment of the content display selection system according to the present invention.

FIG. 4 is a flowchart showing an operation in the second embodiment of FIG. 3;

[Explanation of symbols]

 Reference Signs List 10 control device 12 display device for displaying image and reproducing sound 14A first recognition device for performing voice recognition 14B second recognition device for performing image recognition 16 link information storage unit 18 auxiliary information display control unit 20 content description text 22 server 24 Guide information display controller

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 15/00 G10L 3/00 551P

Claims (5)

[Claims] 1. A content display selection system, comprising: a recognition device including at least voice recognition and image recognition; a display device for displaying content; and control means for controlling the recognition device and the display device in association with each other, The descriptive text describing the content to be displayed includes “category” indicating the type of the recognition device, “recognition candidate data” that is a candidate for a user selection item recognized by the recognition device, and The link destination “link information” is described as recognition information, and the control unit sends a “category” of the display content displayed on the display device to the recognition device,
The “recognition candidate data” is sent to the recognition device so that the “recognition candidate data” displayed on the display device can be recognized by the recognition device, and the corresponding link information is received in response to the recognition result by the recognition device. And controlling the display device to change the content displayed on the display device to the content indicated by the corresponding link information. 2. An "auxiliary information" for displaying a supplementary character string related to the recognition candidate data in the description text.
Is described as the recognition information, the control means sends the "auxiliary information" to the display device, and the display device displays a supplementary character string on the recognition candidate data on a screen by the "auxiliary information". The content display selection system according to claim 1, wherein the content display selection system is configured as follows. 3. The description text describes “guide information” indicating a recognition method included in the recognition device as the recognition information, and the control unit sends the “guide information” to the display device, The content display according to claim 1, wherein the display device is configured to indicate a recognition method included in the recognition device on a screen by the "guide information" and provide a guide of the recognition method to a user. Selection system. 4. The content display selection system according to claim 1, wherein said image recognition recognizes at least one of a gesture, a sign language, a facial expression, and a handwritten pattern. 5. A content including a description for giving a category of a recognition method by voice or image so that a user's intention can be recognized by at least one of a plurality of recognition methods including voice recognition and image recognition. A content recording medium recorded in a readable state.