JPH11305983A - Voice controlled image screen display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To change the image map of a system existent at present into that of voice interactive type in a system for providing voice control image screen display. SOLUTION: This system is composed of an image recognizing part 5 for recognizing features on an image while referring to image data information, keyword link setting part 6 for setting the combination of a keyword expressing the features of the image and a data link, grammar generating part 7 for generating a grammar for voice recognition from the keyword, voice recognizing part 9 for recognizing the input of voice according to the grammar and making it into text data, and input evaluating part 8 for extracting/evaluating the keyword from the text data based on the grammar and the combination of the keyword and data link and performing an instruction to an image screen display part so as to extract document data at the destination of link and to display them when there is the data link corresponding to the keyword.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for providing a voice-controlled image screen display, and to changing the image map of an existing system to voice interactive.

[0002]

2. Description of the Related Art Conventionally, a method of controlling a video (image) screen display has used a pointing device such as a keyboard or a mouse. FIG. 5 shows a conventional configuration for browsing pages on the WWW network using a personal computer or the like. WWW pages are hypertext markup languages
The HTML document is described as a (HTML) document, and the video screen display unit (browser) 11 accesses the WWW network 12 to retrieve and interpret the HTML document and display the HTML document on the video screen display unit (browser) 11. When the user clicks the link display on the video screen with the mouse 13, the video screen display unit (browser) 11 receives the input, accesses the WWW network 12, extracts the HTML document of the linked page, and outputs the HTML document. (Browser) 11. In this way, the user can view the pages of the WWW network 12 one after another.

In this case, a space for operating a keyboard and a mouse is required, and there is a restriction that the user must always use a hand to perform an operation. Voice control image screen displays, including voice recognition systems, are well known, but use language-dependent voice recognition prepared in advance, and do not render existing non-voice recognition systems speech-based. As a technique for solving this, in Japanese Patent Application Laid-Open No. 8-335160, a grammar of voice input is created by recognizing a data link associated with a video screen display, and the recognized text data is evaluated to control the video screen display. A method has been proposed. Take the example of the World Wide Web (WWW) network,
It interprets documents written in Hypertext Markup Language (HTML) and creates grammar for speech recognition.

[0004]

In the image screen display, the display is often controlled by pointing to a figure or a still image displayed on the screen. Clicking a button indicated by a picture with a mouse displays a detailed description or a photo. In the case of performing such an image screen display, using a WWW as an example, an image data file to be a data link and an anchor is specified in an HTML document.
The image data file is so-called image data such as GIF or JPEG. Conventionally, data links described in documents can be extracted and grammar for speech recognition can be created, but grammarization must be performed so that images specified in descriptions in documents can be easily input. Is not taken into account. The HTML tag has an attribute of text data that substitutes for an image, but is not displayed on the screen if the image is displayed, and is not necessarily a word that can be naturally spoken by looking at the image.

[0005] In image screen display, display is often controlled by pointing to a certain part of a figure or a still image displayed on the screen. When a mouse clicks on a place on the map on the screen displaying the map of the facility, a detailed description of the place or a photograph is displayed. To perform such an image screen display, specify the image data file and map in the HTML document, taking WWW as an example. The image data file is so-called image data such as GIF or JPEG. The map describes correspondence information between a coordinate range in the image and a data link when the range is selected.

In the conventional method, a data link described in a document can be extracted and a grammar for voice recognition can be created. However, the coordinates of image data specified by the description in the document can be easily set by the user. It does not consider grammarization in a form that can be entered. Even if the map data is interpreted and all the coordinate values related to the data link are grammarized, it is difficult for the user to determine the coordinate value of the area to be input by looking at the image on the image screen and to utter. It is natural. Therefore, it has been a problem to make the display control by selecting a graphic or a still image on the image screen or pointing a certain portion of the graphic or the still image into a voice interactive type so that the user can easily use it. Accordingly, an object of the present invention is to recognize the contents indicated by image data, create a grammar in the form of words that can be easily specified by a user, and to make the image screen display a voice interactive type.

[0007]

In order to achieve the above object, the present invention provides an image screen display unit for extracting display document data from a data connection, interpreting the display document data, and displaying the image data on an image screen, and analyzing the display document data. A document analysis unit that retrieves image data information in a document and data link information associated with the image data information, an image recognition unit that recognizes features on the image by referring to the image data information, and a keyword that represents the features of the image A keyword / link setting unit that sets a combination of a keyword and a data link; a grammar generation unit that generates a grammar for voice recognition from the keyword; a voice recognition unit that recognizes voice input according to the grammar and converts the input to text data; Extract keywords from the text data based on the combination of keywords and data links Provide an audio control image screen display system comprising an input evaluation unit for instructing the image screen display unit to retrieve and display the linked document data from the data connection if there is a data link corresponding to the keyword after evaluation. .

Further, the present invention provides an image screen display unit for extracting display document data from a data connection, interpreting the display document data, and displaying the image data on an image screen, analyzing the display document data, and displaying image data information in the document and the image data. A document analysis unit for extracting map data information of an upper region and a data link, a map extraction unit for extracting a combination of a region and a data link in image data with reference to the map data information, and referring to the image data information. An image recognition unit for recognizing a region on an image and its characteristics, a keyword / link setting unit for setting a combination of a keyword and a data link representing the characteristics of the region in the image, and generating a grammar for speech recognition from the keyword Grammar generator for performing speech recognition based on the grammar and recognizing speech input into text data Extracting and evaluating a keyword from the text data based on the combination of the grammar, the keyword, and the data link, and if there is a data link corresponding to the keyword, retrieves and displays the linked document data from the data connection. An audio control image screen display system comprising an input evaluation unit for instructing an image screen display unit is provided.

[0009]

DETAILED DESCRIPTION OF THE INVENTION The present invention provides a video (image) screen display control method based on voice recognition for an image screen display system which does not assume control based on voice recognition. In particular, in image data specified in an image screen display document, conventionally, a user clicks a part of an image displayed on an image screen with a pointing device such as a mouse, and selects the image based on the selected coordinate data or the clicked coordinate data. This enables control by voice recognition in a scene where the image screen display device is performing display control. A browser that accesses an embodiment of the present invention and displays a screen on a world wide web (WWW) network of the Internet, which is a well-known field,
The case where voice control by voice recognition is used for WWW operation will be described below.

[0010] Fig. 5 shows a conventional configuration for browsing a WWW network page using a personal computer or the like. The function of jumping to another link destination according to a clicked area of an image displayed on a browser in an HTML document is illustrated. Call it a map. In an HTML document using an image map, an image data file specification and a map specification are described. The image data file is image data in a standard format such as GIF or JPEG. The specification of the map is to associate a certain area on the image with the link destination,
The coordinates, range (shape / size), and link destination are described for each area. When the user clicks the mouse on the displayed image, if the clicked position is within the designated area of the link, the link destination page is displayed. HTML can also describe image data files as anchors to links. In this case, by clicking the image displayed on the image screen, the user can jump to the link destination.

The above-described image map and the link using the image as an anchor will be described below with reference to FIG. 1 according to an embodiment of the present invention. A video screen display unit (browser) 1 shown in FIG. 1 is connected to a data connection WWW network 2 to take out an HTML document, and displays a page screen on the video screen display unit 1. The document analysis unit 3 connects to the video screen display unit (browser) 1 to read and analyze the HTML document. If the HTML document has an image map description, the map data specified according to the description and the image in the HTML document. The data and the map data are sent to the map extraction unit 4 and the image data is sent to the image recognition unit 5. If there is a link description with an image as an anchor, the link destination and size information are sent to the map extraction unit 4.
The image data is sent to the image recognition unit 5 respectively.

In the map extracting section 4, from the map data,
A list (map list) of regions and link destinations in the image is created and sent to the keyword / link setting unit 6. This area is represented by values such as position, shape, and size. In the case of a link using an image as an anchor, the entire image is registered in the map list as one rectangular area. The image recognition unit 5 recognizes a region that can be selected in the image data. Recognize character strings (including one character), figures, and objects represented in images, and for each recognized object, its area, position, shape (name), size, color, and character string The character string is taken out, an image list is created, and sent to the keyword / link setting unit 6. The shapes, logos and images to be recognized are
It can be registered in the recognition image database 5a,
The character database for recognition 5b is used for character recognition. In the case of a link using an image as an anchor, one region is extracted for each image and registered in the image list.

The keyword / link setting unit 6 compares the map list from the map extracting unit 4 with the image list from the image recognizing unit 5 and sets a keyword and a link destination for each area. An area of the image list that matches the area of the map list is extracted, and the position, shape (name), size, color, character string, and the like regarding this area are set as a keyword to the corresponding link destination using words. Here, the set keywords are set with reference to the keyword dictionary 6a, taking into account the relative relationship between the respective regions based on the information in the image list. For example, when three figures are arranged side by side, keywords “left”, “middle”, and “right” are set for the positions, respectively. Regarding the size, if the figure of size 10 and the figure of size 5 are arranged side by side, the figure of 10 is "large" and the figure of 5 is "small", but the figure of size 10 and the figure of 20 are If they are lined up, the keywords are set as "small" for 10 figures and "large" for 20 figures. In this way, a keyword / link list of a set of a keyword group and a link destination for each area is created. The list generated by the keyword / link setting unit 6 is sent to the grammar generation unit 7 and the input evaluation unit 8.

The grammar generation unit 7 generates a grammar that is a candidate for speech recognition based on the keywords in the keyword / link list. This grammar can be described and compiled using syntax rules. The grammar data generated by the grammar generation unit 7 is sent to the input evaluation unit 8 and the speech recognition unit 9. The speech recognition unit 9 recognizes the voice uttered by the user based on the grammar generated by the grammar generation unit 7 and sends the recognition result to the input evaluation unit 8 as text data. The voice recognition unit 9 includes an input instruction unit 9a. Upon receiving the grammar data, the input instructing unit 9a outputs an instruction to prompt the user to input, and receives the re-input instruction information from the input evaluation unit 8 and outputs an instruction to prompt the user to re-input. The output of the instruction is Text To Spee
The sound of the channel or the display on a part of the image screen or a separately provided display device can be performed.

The input evaluation unit 8 analyzes the text data sent from the speech recognition unit 9 according to the grammar, and checks the text data against a keyword group in a keyword / link list. If the link destination is uniquely determined from the extracted keywords, it is determined that the link destination has been selected, and the selection of the link destination is input to the video screen display unit (browser) 1. Upon receiving the input, the video screen display unit (browser) 1 accesses the WWW network 2, extracts the HTML document of the linked page, and displays it on the video screen display unit (browser) 1. In this way, the user can view the pages of the WWW network 2 one after another. When the link destination is not uniquely determined from the keywords extracted by the input evaluation unit 8, or when an inappropriate combination of keywords is extracted, the voice recognition unit 9 is requested to input again. The voice recognition unit 9 issues a re-input instruction to recognize the voice of addition or correction of the user, and sends text data to the input evaluation unit 8. The input evaluation unit 8 analyzes the text data sent from the speech recognition unit 9 according to the grammar, and extracts and evaluates an additional keyword or a corrected keyword. This can be repeated until the link destination is uniquely determined.

Next, an embodiment of the present invention in which the simple image map shown in FIG. 2 is of a voice interactive type will be described below. The X axis and the Y axis in FIG. 2 indicate the coordinates of the image that increases from the upper left to the lower right. The rectangular range of (Xp, Yp) is an image screen display image of this image data. The dotted line in the figure is for making the coordinate designation easy to understand, and is not included in the display image. In the image map, the round area of “A” goes to link 1; the square area of “i” goes to link 2;
It is assumed that the round area of “U” is described so as to link to the link destination 3. However, "Please choose"
Since a certain square is a title, no link is described.
Table 1 shows an example of a map list extracted by the map extracting unit 4.
become that way. On the other hand, an example of the image list extracted by the image recognition unit 5 is as shown in Table 2.

[0017]

[Table 1]

[Table 2]

The image list area is used to match the map list area. The position is the coordinates of the center of the recognized area, the shape is the name of the figure or object being drawn, the size is the horizontal and vertical size of the rectangular range including the recognized area, and the colors are R and G of the main colors of the area. B: Representation at 256 gradations, and the character string is recognized character string data. The keyword / link setting unit 6 sets a keyword for each area from the map list and the image list, and creates a keyword / link list. Here, the keywords are set based on the information of the image list, taking into account the relative relationship between the respective regions. For example, in the region numbers G1 and G3, the positions are in an up-down relationship, so that the keywords "up" and "down" are given. Even in the same square, the aspect ratios are clearly different. Number G2 ・ G3
・ Because G4 is lined up sideways, "Left", "Middle"
The "right" keyword is given respectively. For the color, a color name suitable for identifying the color of each area is given as a keyword.
If there is an area with a similar color, keywords such as "dark" and "light" are added. Table 3 shows an example of the keyword / link list.

[0019]

[Table 3]

The grammar generation unit 7 arranges the keyword group into a grammar form in which the user inputs speech, and an example of the syntax is as follows. <Input statement> = <position>? <Color>? <Name><position> = top | bottom | left | right | middle <color> = green | yellow | blue | red <name> = circle | square | | Square…

The speech recognition unit 9 recognizes speech according to the grammar and sends the speech to the input evaluation unit 8 as text data. The input evaluator 8 analyzes the input text data based on the grammar data and the keyword / link list, and determines whether a link destination can be determined from a keyword in the text data. For example, if "yellow circle", "a", "left circle" or the like is input, it indicates area 2 and the link destination can be determined to be link destination 1. However, if "circle" is simply input, area 2 I can not judge whether it is 4 or not. In such a case, a message prompting an additional keyword such as "Which circle?" Is output from the voice recognition unit 9 to acquire the additional keyword. When a combination of keywords having no corresponding area, such as “blue circle”, is input, a message indicating that there is no corresponding area is output, and re-input is prompted. "please choose"
When there is no link destination as in the above, a message indicating that there is no link is output, and the user is prompted to input again. Here, if “square” is simply input, the area 1 and the area 3 can be considered, but no link destination is specified in the area 1. In such a case, the area 1 is regarded as a display not included in the options,
Speaking of the “square”, it is determined that the area is 3, and the link destination 2 can be called.

The shape of the image extracted by the image recognition unit 5 is a circle,
It is extracted not only by a simple figure such as a square, but also by the name of an object represented in the figure. Although the three pictures (A) to (C) shown in FIG. 3 are all rectangular areas of the same size in the map list, the image recognizing unit 5 refers to the recognition image database 5a to generate the respective areas. , "Notes", "Newspapers", "Calendars"
Recognize and distinguish by names such as In the map list of FIG. 3D, two circular areas are provided, but the image recognition unit 5 recognizes the area as one “snowman”. Therefore, the user can make a selection based on the name of an object in the viewed image.

The keyword / link setting section 6 has a keyword dictionary 6a, and stores the association knowledge listed in Table 4 in the keyword dictionary 6a. If there is an image in which a yellow circle as shown in FIG. 4 is drawn, the keyword / link setting unit 6 sets “egg yolk” if the background area is white, “sun” if blue, and “black” if black. For example, a keyword that satisfies the condition of associative knowledge, such as “month”, can be set. As described above, the application to the image screen (browser) 1 connected to the WWW network 2 has been described as an example. However, even in a display on a terminal screen in a local network system or a stand-alone image screen display system, a fixed display image In a scene where display control is performed by selecting an area, the display control can be effectively implemented for voice interactive type. The format and unit of expression such as region, position, size, color and the like shown in the present embodiment are also examples for explanation, and the scope of the present invention is not limited to this.

[0024]

[Table 4]

[0025]

According to the voice control image screen display system of the present invention, the contents indicated by the image data are recognized, and the grammar is created in the form of words that can be easily specified by voice by the user based on the correspondence with the map data link. The image screen display can be controlled by referring to the data link from the voice recognition result. That is, the keyword corresponding to the link region of the image can be set by a word representing the feature of the image in that region. Thus, conventionally, when the user clicks a part of the image displayed on the image screen with a pointing device such as a mouse, and the image screen display device performs display control based on the image selection or the clicked coordinate data. In, display control can be performed by designating a region of the image using words that can be naturally designated by the user while viewing the display image. That is, by providing an audio control image screen display system corresponding to an image map, hand-free navigation for an image screen display such as a world wide web network or an interactive television display becomes possible.

The voice control image screen display system of the present invention can be used in combination with the conventional image screen display control method. Can be added.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an embodiment of a voice control image screen display system according to the present invention.

FIG. 2 is an example of an image of an image map according to an embodiment of the present invention.

FIG. 3 is an example of an image recognized by an image recognition unit according to an embodiment of the present invention.

FIG. 4 is an example of an image according to an embodiment of the present invention.

FIG. 5 is a configuration diagram of an example of a conventional image screen display device.

[Explanation of symbols]

 1 video (image) screen display unit (browser) 2 data connection (www network) 3 document analysis unit 4 map extraction unit 5 image recognition unit 5a recognition image database 5b recognition character database 6 keyword / link setting unit 6a keyword dictionary 7 Grammar generation unit 8 Input evaluation unit 9 Voice recognition unit 9a Input instruction unit

────────────────────────────────────────────────── ─── front page continued (51) Int.Cl. ⁶ identifications FI G06T 7/00 G10L 3/00 521C G09G 5/00 530 551G G10L 3/00 521 551Z 551 G06F 15/62 321A 15/70 330Q

Claims (6)

[Claims] 1. An image screen display unit for extracting display document data from a data connection, interpreting the display document data, and displaying the display document data on an image screen, analyzing the display document data and associating the image data information in the document with the image data information. A document analysis unit for extracting data link information to be obtained, an image recognition unit for recognizing a feature on an image by referring to the image data information, and a keyword / link setting for setting a combination of a keyword and a data link representing the feature of the image A grammar generation unit that generates a grammar for speech recognition from the keyword; a speech recognition unit that recognizes speech input according to the grammar and converts it into text data; and a text based on a combination of the grammar, the keyword, and a data link. Extract and evaluate keywords from data and match keywords Voice control picture screen display system characterized by comprising the input evaluation unit and instructing said image screen display unit so that Tarinku to retrieve and display document data destination from the data connection, if any. 2. An image screen display unit for extracting display document data from a data connection, interpreting the display document data, and displaying the display document data on an image screen; analyzing the display document data to obtain image data information in the document and an area on the image; A document analysis unit that extracts map data information of a data link; a map extraction unit that extracts a combination of a region and a data link in image data with reference to the map data information; An image recognition unit for recognizing a region and its feature; a keyword / link setting unit for setting a combination of a keyword and a data link representing a feature of the region in the image; and a grammar generation for generating a grammar for speech recognition from the keyword A voice recognition unit that recognizes voice input according to the grammar and converts the voice input into text data; Extracting and evaluating keywords from the text data based on a combination of grammar, keywords and data links, and displaying the image screen so that if there is a data link corresponding to the keyword, the linked document data is extracted from the data connection and displayed. A voice control image screen display system, comprising: an input evaluation unit for instructing the unit. 3. The voice-controlled image screen display system according to claim 1, wherein the image recognition unit has at least one of a recognition image database and a recognition character database, and the recognition image database. Are the shapes, logos, people,
Recognize other objects and give their names as features of the area in the image, the recognition character database recognizes a single character or character string in the image, and converts the character or character string to the area in the image. An audio-controlled image screen display system, characterized in that it is provided as a feature. 4. The voice control image screen display system according to claim 1, wherein the keyword / link setting unit holds a keyword dictionary and compares characteristics of a plurality of image areas displayed on the image screen. A voice control image screen display system, wherein keywords are provided based on a relative relationship between a plurality of regions. 5. The voice control image screen display system according to claim 1, wherein the input evaluation unit extracts and evaluates a keyword from the text data based on a combination of a grammar, a keyword, and a data link, and evaluates the keyword. If there is no data link destination corresponding to, the voice recognition unit requests re-input, the voice recognition unit outputs an input instruction when receiving grammatical data, and outputs a re-input instruction in response to the re-input request of the input evaluation unit. A voice-controlled image screen display system. 6. The voice control image screen display system according to claim 4, wherein the keyword dictionary has associative knowledge regarding at least one of a color and a shape, and the keyword / link setting unit includes:
A voice control image screen display system, wherein a keyword associated with at least one of a color and a shape between a plurality of image areas is set.