JP3170103B2 - Object pointing method by compound form - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user interface such as a graphic editing system mounted on an OA device such as a personal computer (personal computer), a workstation, or a word processor (word processor). The present invention relates to a method of pointing an object by a form.

[0002]

2. Description of the Related Art A conventional object pointing method is disclosed in Kitahara et al., "Study on Spoken Word Receiving Method in Information Retrieval System by Voice Input" (Proceedings of the Acoustical Society of Japan, Spring Meeting, 3-5-7).
As can be seen in 19911.3) and the like, an instruction of an object using only sound or an instruction of a pointing device such as a mouse within a display range of the object (mouse event)
An object that recognizes a target object is known.

When a figure is designated, BABolt "P
ut that there "(Computer Graphics, 14,3,1980) pp2
As disclosed in JP-A-62-270, in addition to giving instructions by uttering demonstrative pronouns accompanied by gestures, before performing graphic editing, a name is given to an object drawn on the screen in advance, and this name is hereinafter referred to as this name. There was a method such as using.

[0004]

In the above prior art,
When an object or an object to be instructed on the screen exists discretely, a voice instruction such as “this” or “this square” is performed while pointing the object on the screen with a finger or a pointing device. That is enough to identify the object. However, the objects on the screen are close to each other,
Alternatively, in the case of overlapping, it may be difficult to uniquely identify the target object with a finger or the like, and it may be necessary to supplement the instruction with a more complicated voice instruction.
On the other hand, a more complicated voice instruction has a problem that the possibility of erroneous recognition increases.

An object of the present invention is to enable a user to input information naturally without being conscious of the form of use of media, and to use a plurality of input modes for erroneous recognition, which is a major problem in voice recognition. Decrease, shape editing,
An object of the present invention is to provide a method of pointing an object that is natural and easy for a user to use in image editing and the like.

[0006]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, a method of pointing an object by a compound form according to the present invention comprises:
When instructing one of a plurality of display objects displayed on the screen of the display device as an object, a pointing instruction with a finger or a pen and an instruction by voice are used in combination, an object instruction method in a complex form, The mutual positional relationship between the plurality of display objects is obtained, a display object knowledge table representing the positional relationship is created, and the speech recognition result for the instruction of the display object by the voice including the positional relationship is stored in the display object knowledge table. In the light of the content, an object candidate indicated by the voice is obtained, position information based on the pointing instruction is obtained, and an object is specified based on the position information based on the pointing instruction and the object candidate. It is.

[0007] Also, a composite form input device according to the present invention is provided
Information display means for displaying information; pointing means for indicating a position on the display screen of the information display means using a finger or a pen; and voice recognition means for recognizing an instruction by voice; A combined form input device for instructing one of a plurality of display objects on the display screen as a target object by using together with a voice instruction, wherein at least the names of the plurality of display objects displayed on the display screen Means for storing a word dictionary including a word representing the word and a word representing the mutual positional relationship between the plurality of displayed objects, means for determining the mutual positional relationship between the plurality of displayed objects, and display object knowledge representing the positional relationship Means for storing a table, means for obtaining position information based on the pointing instruction,
An object that specifies the name and the positional relationship of the display object recognized by the voice recognition unit with reference to the display object knowledge table and obtains the object candidate, and specifies the object from the object candidate based on the position information. And specifying means.

[0008]

The present invention has a knowledge table which stores a mutual positional relationship between a plurality of display objects displayed on a screen. When a user designates an object by voice or pointing gesture, voice information and pointing gesture are taken in. It is. For voice input, for example, morphological analysis and syntax analysis of voice information are performed as a voice recognition technique, and the name of the graphic of the object and the positional relationship with other graphics are extracted based on the analysis result. By comparing the speech recognition result with the graphic knowledge table, the object candidate is narrowed down. Further, the target object is uniquely specified from the target candidate group in light of the position coordinates input by the pointing gesture.

[0009] Further, even if the object candidate has a large likelihood calculated by the voice information analysis means, there is a possibility that a voice recognition error may occur. In such a case, the target is considered in the range specified by the pointing gesture. Since the object candidate is excluded, an object recognition error due to a speech recognition error can be prevented.

When a plurality of object candidates are recognized by one operation, and it is necessary to further narrow down the object, the object candidate enlargement means maintains the positional relationship and displays the object candidates in an enlarged manner. Then, the user can narrow down the candidates by re-inputting the voice, the pointing gesture, and the like.

[0011]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the present invention. Here, description will be made assuming a graphic editing system. However, the present invention is not limited to this system,
Application to general image processing systems such as an AD system is possible.

The system shown in FIG. 1 includes an information processing device 1, a main storage device 2, a touch panel 5, a control device 3 for the touch panel 5, a display 4, a display control device 6 for controlling the display 4, a microphone 8, and a microphone 8. And an A / D converter 7 for converting an input signal into a digital signal, and a disk 9 as an external storage device for storing various programs and data.

The system program 11 on the disk 9
The figure editing program 12, the sound recognition program 13, the pointing coordinate reading program 14, the object instruction program 15, the sound standard pattern data 16, the word dictionary 17, and the display object knowledge table creating program 18
When the system is started, it is loaded into the main storage device 2.

As shown in FIG. 5, the sound recognition program 13 comprises a voice input program 131 and a feature extraction program 132.

As shown in FIG. 10, the object pointing program 15 includes a pointing area recognition program 151,
It comprises a dictionary matching program 152 and an object candidate expanding program 153.

As shown in FIG. 11, the word dictionary 17 used at the time of starting the dictionary matching program 152 is composed of a word number 111, a synonym representative word 112 and synonyms 113 having similar meanings belonging to each word number slot. Become. For example, the synonyms "Yen" and "Maru" have the representative word "Yen", and the synonyms "Inside", "Inside", "Middle" and "Middle" have the representative word "Inside". I have. The word number 111 is the same as the relationship description number stored in the display object knowledge table for words (for example, inside and outside) representing the mutual positional relationship of the display objects defined in the display object knowledge table (FIG. 11) described later. It is.

FIG. 19 shows various buffer memories on the main storage device 2 used in the processing in this embodiment. Each of these buffer memories will be described in the processing to be used.

FIG. 2 is an example of a graphic editing screen displayed on the display 4 through the graphic editing program 12 loaded in the main storage device 2. In this example, the screen has a width of 1000 and a height of 700. In the figure, based on the circle drawing table, the rectangle drawing table, and the triangle drawing table stored in the main storage device 2 by the graphic editing program 12, seven circles, one rectangle,
And one triangle are drawn in graphic mode.

As shown in FIG. 3, a circle drawing table TB1
Are table number E = 801, circle number 100, center x coordinate 101, center y coordinate 102, radius 103, and color number 1
04, the rectangle drawing table TB3 has a table number E = 802, a rectangle number 200, and x at the upper left vertex.
The coordinate 201 includes a coordinate 201, a y coordinate 202, a portrait 203, a landscape 204 and a color number 205. The triangle drawing table TB3 has a table number E = 803, a triangle number 3
00, x coordinate 3011 of vertex coordinate 1, y coordinate 3012 of vertex coordinate 1, x coordinate 3021 of vertex coordinate 2, vertex coordinate 2
, A vertex coordinate 3 x coordinate 3031, a vertex coordinate 3 y coordinate 3032, and a color number 304. The color number is “100” when the figure color is white.
1 "," 1002 "for black," 100 "for red
3 ". The circle drawing table T in FIG.
In the example of B1, the center (280, 34
0), radius 50, white circle, center (390, 230) as circle number C2, radius 50, black circle, circle number C2
3 as center (390, 340), radius 20, circle of white color, circle number C4 as center (390, 340), radius 1
0, circle of white color, center as circle number C5 (530, 27
0), a circle having a radius of 40, a white circle, and a circle having a center (530, 270), a radius of 60, and a white circle as a circle number C6 are drawn. In the example of the rectangle drawing table TB2, the coordinates of the upper left vertex are (140,
460), a rectangle 120 vertically long, 250 horizontally long, and white in color are drawn. Similarly, the triangle drawing table TB
In the example of No. 3, the vertex coordinates (62
0, 520), (550, 660), (680, 63)
0), a triangle whose color is white is drawn. The XY coordinates on the screen have the origin at the upper left of the screen.

At this time, it is assumed that the numbers of circles, rectangles, and triangles drawn in the drawing table are stored in the buffer memories F1, F2, and F3 on the main storage device 2. That is, in the above example, "6" is stored in F1, "1" is stored in F2, and "1" is stored in F3. Further, the value of F1 + F2 + F3 is stored in the buffer memory W on the main storage device 2. In the case of this embodiment, W = 8
Becomes Further, the serial numbers of the entire screen display are numbered from 1 to W in the order of circle, rectangle, and triangle. For example, a figure with a serial number = 1 is a figure with a circle number “C1”, and a figure with a serial number = 7 is a figure with a rectangle number “R1”.

Further, a display object knowledge table storing the positional relationship between the display objects is provided on the main storage device 2. In this display object knowledge table, for example, as shown in a table NT of FIG. 4, a positional relationship between arbitrary two figures A and B is defined by using a relation explanation number as follows. That is, “6050” when the graphic A is on the right side of the graphic B, “6040” when the graphic A is on the left side, “7040” when the graphic A crosses the graphic B, and the graphic A is included in the graphic B. If the figure A includes the figure B, it is set to "8042" (included), and if the figure A includes the figure B, it is set to "8041" (included). Although only one relationship description number is shown between FIG. A and FIG. B in the figure, a plurality of relationship description numbers may be used. For example,
In the case of right side and intersection, two related explanation numbers “6050” and “7040” are stored. The display object knowledge table NT can store not only the positional relationship between the display objects but also the attributes of the display objects, for example, the relationship between the vehicle and the window (the window of the vehicle).

FIG. 18 shows a processing flow of the display object knowledge table creating program. This process is executed when a figure is input and created on the screen.

First, the maximum value Xmx1 and the minimum value Xmn1 of the X coordinate of the coordinate values of all the contour points of each figure are extracted (s181), and the Xb in the range of Xmn1 <Xb <Xmx1 is converted to the coordinate value. Is extracted, and its minimum value Ymn1 (Xb) and maximum value Ymx1
(Xb) is obtained (s182). Therefore, for all combinations of displayed graphics, the Xm
n1 <Xb <Xmx1, Ymn1 (Xb) <Yb <Ym
With respect to the range of x1 (Xb), an inclusion relation (including a cross relation) is determined in consideration of the presence / absence of common coordinates of the two figures (s183). Further, the average value Xni of the X maximum value and the minimum value of each figure and the average value Yni of the Y maximum value and the minimum value are obtained.
Is calculated (s184), (Xni, Yni) is compared for all of the displayed graphics, and the mutual positional relationship is extracted (s185). The result thus obtained is stored in the display graphic knowledge table NT of FIG.

Although the processing in FIG. 18 is performed on all the figures at once, each time a new figure is added, the positional relationship between the additional figure and the existing figure is obtained. It is also possible.

FIG. 14 shows the flow of processing after the graphic editing program 12 is started.

It is assumed that the user designates one of the objects displayed on the screen. First, the information processing device 1
Starts the sound recognition program 13 on the main storage device 2 (s301), and further starts the pointing coordinate reading program 14 (s302). When the sound recognition program 13 is started, first, the voice input program 1
31 (FIG. 5) is activated. The user touches the touch panel 5
Above, an object is pointed with a finger or a pen or the like (FIG. 6), or only a sound is generated using the microphone 4 (an example of utterance “rectangular”).
(FIG. 7) (s30)
3). If an instruction of only pointing input or voice input by a finger or the like is insufficient, the target object is indicated by voice (“inner circle”) using the microphone 4 while simultaneously indicating the target object on the touch panel 5 ( (FIG. 8). When a voice input is performed, “1” is written into the voice input identification buffer memory A on the main storage device 2. If no voice input is made, “0” is written to the buffer memory A. When the pointing is input, “1” is written into the pointing input identification buffer memory B on the main storage device 2. When no pointing is input, “0” is set in the main storage device 2.
The data is written to the upper buffer memory B.

First, a case where two voice inputs and a pointing input are performed simultaneously as shown in FIG. 8 will be described.
In this case, when a user's voice is input (s101),
According to the voice input program 131, this voice signal is A
After being taken into the / D conversion device 7 and converted into a digital signal (s102), it is sent to the main storage device 2. Subsequently, a feature extraction program 132 (FIG. 5) is started, and the digital signal is described as a feature vector with a frame period of 10 ms, for example, as described in Saito and Nakata, "Basics of Speech Information Processing" (Ohmsha, Showa 56). Is converted to a time series of LPC cepstrum coefficients (s103). Here, the frame period is not limited to 10 ms, but may be 20 ms, 30 ms.
ms and the like can be set arbitrarily. Also, the feature vector is not limited to the LPC cepstrum coefficient, but may use an output of a band-pass filter or the like.

On the other hand, the pointing coordinate reading program 14 in the main storage device 2
Is activated by the information processing apparatus 1 at the same time as the activation (s302), and parallel processing is performed. An example of the flow of processing after starting the pointing coordinate reading program 14 will be described with reference to FIG.

First, the buffer memory P in the main storage device 2
And Q are reset to zero (s201, s202). The program captures contact coordinates through the panel control device 3 at regular time intervals while the user's fingertip or pen touches the touch panel 5 (s203) (s20).
4), P is incremented each time coordinates are fetched, and further, as a pointing area table of the main storage device 2,
The fetched x coordinate is stored in the array memory X [P], the y coordinate is stored in the array memory Y [P], and the coordinate input time is stored in the array memory T [P].
Write to [P] (s205). As shown in the table PT of FIG. 9, the pointing area table has a coordinate number 200, an input time 201, an x coordinate 202, and a y coordinate 203.
The input time, the x-coordinate and the y-coordinate data are stored in a coordinate number “1” in the input order at regular intervals.
It is stored from. In the above example, the coordinate data is stored every 100 ms. When a certain time To elapses after the fingertip or the pen leaves the touch panel 5, the writing is terminated (s203).

Returning to FIG. 14, when the pointing input or the voice input by the user is completed, the object instruction program 15 (FIG. 10) in the main storage device 2 is started (s30).
4). First, the value of the buffer memory A is checked (s
305), when the buffer memory A is "1" (that is, when voice input is performed), the dictionary matching program 152 is started first (s306), and then the value of the buffer memory B is confirmed (s307). When the buffer memory B is "1" (when the pointing input is performed), the pointing area recognition program 151 is started (s308). When the buffer memory A is "0" (when no voice input is performed), the pointing area recognition program 151 is immediately started (s3).
11).

In the example of FIG. 8, since the buffer memory A is "1" and the buffer memory B is "1", both the pointing area recognition program 151 and the dictionary matching program 152 are started.

According to FIG. 16, dictionary matching program 1
The processing of 52 will be described. Matching between the previously obtained feature vector and the sound standard pattern data 16 is performed (s161). This is performed, for example, by the method described in Kitahara et al., “Study of Spoken Language Receiving Method in Information Retrieval System by Voice Input” (Acoustical Society of Japan, 3-5-7, Hei 3). Is converted to a string. In the above example, this character string is “Uchigawa no En”. Further, the character string is obtained by using a conventional method, for example, the longest matching method described in Kana-Kanji conversion by computer (NHK Technical Research, 25, 5, 48). Morphological analysis (s1
62) Further, matching with the word dictionary 17 (FIG. 11) is performed, and individual words are aggregated into representative words. As a result of giving word numbers, the maximum likelihood value of the speech recognition result (“inside”) , Noun, 8042), ("no", case particle, 9
00), (“circle”, noun, 701). Next, a relation explanation number in a circle column (one vertical column of the graphic A) of the display graphic knowledge table NT (FIG. 4) in which the positional relation between the display objects is stored in the main storage device 2;
The word number in (“inside”, noun, 8042), that is, “8042” is matched (s163), and a candidate object group is extracted (s164). In this example, the circle number “C4” and the circle number “C5” in the circle drawing table are extracted. Further, the number of object candidates is stored in the buffer memory G on the main storage device 2. In the present embodiment, G = 2.

Next, the processing of the pointing area recognition program 151 will be described with reference to FIG. The minimum value XMn and the maximum value XMx of the X coordinate values stored in X [1] to X [P] on the main storage device 2, and Y [1] to Y
The minimum value YMn and the maximum value YMx of the Y coordinate values stored in [P] are calculated (s151), and the coordinates (Xnt, Xnt, Xn, XMx, XMx) and the average value of YMn, YMx indicate.
Ynt) is obtained (s152). Next, the coordinates (Xnt,
The distance lg between Ynt) and each object candidate in the object candidate group is sequentially calculated (s153), and the object candidate having the shortest distance is uniquely identified as the object (s154).

Returning to FIG. 14, when a plurality of object candidates equidistant from (Xnt, Ynt) are obtained from the object candidate group by the processing of the pointing area recognition program 151 in FIG. 15 (s309). Then, the object candidate enlarging program 153 is activated (s310), the candidate graphic group is enlarged while maintaining the positional relationship, and the object is again instructed by voice or pointing (s303).
The object can be squeezed out of the plurality.

Referring to FIG. 17, the processing of the object candidate group expansion program 153 will be described. First, a window is generated in a certain area of the candidate object group (s171). Subsequently, an enlarged image is displayed on the window while maintaining the mutual positional relationship between the candidate object groups (s172). Therefore, the object is determined by the user's instruction again (s173).
Thereafter, the window is deleted (s174).

In the example of FIG. 8, as a result of speech recognition, the recognition candidate having the highest likelihood is the "inner circle", and circles C4 and C5 are obtained as object candidates. In addition, the pointing area recognition program 151 determines that the circle C4 whose distance from the designated coordinates to the center coordinate of each of the candidate objects in the group of candidate objects is the designated object.

In step s308 of FIG. 14, since the target candidate has already been obtained as a result of speech recognition by the dictionary matching program 152, the pointing coordinates and the center coordinates of the target candidate are used in the pointing area recognition process. A simpler method such as the following can be used without calculating the distance to. That is, it is checked whether or not the center coordinates of each object candidate are included in the vicinity in the X direction (or Y method) designated by the pointing gesture, and if there is one included object candidate. If this is the case, it is determined to be the designated target. If there are two or more, the target object group is enlarged and the instruction is performed again. If the included object candidate does not exist, there is a possibility of erroneous speech recognition, and the next largest likelihood calculated by the speech information analysis means is sequentially represented by a word corresponding to the likelihood. It is determined whether or not the center coordinates of the object are within a range specified by the pointing gesture, and when a candidate existing within the range is extracted, the candidate is uniquely identified as the object.

The range specified by the pointing gesture is, for example, XMn-d1 ≦ Xj ≦ XMx
+ D2 (values of d1 and d2 will be described later) within a range of coordinates Xj. Here, d1 is 1/1 of the horizontal width of the screen.
The length is 0, and here, d1 = 10. D2 is 1/20 of the screen width, and here, d2 = 5. The reason for increasing d1 is that it is considered that a right-handed user indicates the right side of the figure. For left-handed users, the values of d1 and d2 are reversed. Therefore, it is preferable that the values of d1 and d2 can be set by the user. However, if the range specified by the pointing gesture is determined not in the X direction but in the Y direction, that is, YMn-d1 ≦ Yj ≦ YMx + d2
Then, d1> d2 may be set regardless of the dominant arm.

As shown in FIG. 7, when only voice input is performed, "1" is written into the voice input identification buffer memory A on the main memory 2 and "1" is written into the pointing input identification buffer memory B. 0 "is written. When the buffer memory A is “1” and the buffer memory B is “0”, only the dictionary matching program 152 is started,
Matching between the obtained feature vector and the sound standard pattern data 16 is performed as described above, and as a result, the input voice is converted into a character string. In the example of FIG. 7, this character string is “chohokei”. In addition, this string
As a result of morphological analysis using the longest matching method described above and further matching with a word dictionary, morphological information such as (“rectangle”, noun) is obtained. Next, matching is performed between the extracted noun number and the existing table number, and in this example, the table number E = 802 is extracted.
Since only one figure R1 is stored in the table, R1 is selected, and the rectangle R1 is highlighted (for example, blinks) on the screen.

As shown in FIG. 5, when only pointing input is performed, "0" is written into the voice input identification buffer memory A on the main storage device 2, and "0" is written into the pointing input identification buffer memory B. 1 "is written.
Buffer memory A is "0", buffer memory B is "1"
In the case of, the pointing area recognition program 151
Are activated, and X [1] to X
The minimum value XMn of the X coordinate values stored in [P]
And the maximum value XMx, the minimum value YMn and the maximum value YMx of the Y coordinate values stored in Y [1] to Y [P] are calculated, and the average value of XMn and XMx and the average value of YMn and YMx are calculated. The distance between the coordinates (Xnt, Ynt) indicated by and the individual object candidates in the object candidate group is sequentially calculated, and the object candidate having the shortest distance is uniquely identified as the object. Also, (Xn
If there are a plurality of object candidates that are the shortest distance from (t, Ynt), the object candidate enlarging program 153 is started (s310), the candidate graphic group is enlarged while maintaining the positional relationship, and the sound is again reproduced. By giving an object instruction by pointing or pointing, the object can be squeezed out of the plurality of objects.

[0042]

According to the present invention, by providing a table for storing the inclusive relation of a plurality of display objects displayed on the screen and the positional relation between the display objects, it is possible to obtain the audio information input by the user and the position information by the pointing gesture. The target object intended by the user among the plurality of screen display objects can be accurately recognized. If the display object corresponding to the word having the maximum value among the likelihoods calculated in the speech recognition process does not exist within a certain range instructed by the pointing gesture, the speech recognition with the sequentially larger likelihood is performed. By adopting the candidate words, it is possible to reduce the erroneous recognition of the object due to the speech recognition error, and the usability of the user is improved. In addition, when a plurality of object candidates are recognized by one operation, the function of enlarging and displaying the object candidates while maintaining the positional relationship is provided, so that the object can be uniquely designated by only one input information. Even if it is not possible, the object can be uniquely identified quickly and easily.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a graphic editing system according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram illustrating an example of a graphic editing screen according to the embodiment;

FIG. 3 is an explanatory diagram illustrating an example of a data structure of a drawing table according to the embodiment.

FIG. 4 is an explanatory diagram showing an example of a data structure of a display graphic knowledge table in the embodiment.

FIG. 5 is an explanatory diagram showing a module configuration of a sound recognition program in the embodiment.

FIG. 6 is an explanatory diagram of an example of a graphic editing screen showing a state in which an object is designated only by a pointing gesture in the embodiment.

FIG. 7 is an explanatory diagram of an example of a graphic editing screen showing a state in which a target is indicated only by voice in the embodiment.

FIG. 8 is an explanatory diagram of an example of a graphic editing screen showing a state in which a target is indicated using both a voice and a pointing gesture in the embodiment.

FIG. 9 is an explanatory diagram illustrating an example of a data structure of a pointing area table in the embodiment.

FIG. 10 is an explanatory diagram showing a module configuration of an object instruction program 15 in the embodiment.

FIG. 11 is an explanatory diagram showing an example of a data structure of a word dictionary 17 in the embodiment.

FIG. 12 is a flowchart illustrating an example of a flow of processing after activation of the acoustic recognition program 13 in the embodiment.

FIG. 13 is a flowchart illustrating an example of a processing flow after starting the pointing coordinate reading program according to the embodiment.

FIG. 14 is a flowchart illustrating an example of a process flow after the graphic editing program 12 is started in the embodiment.

FIG. 15 is a flowchart illustrating an example of a process flow after the pointing area recognition program 151 is started in the embodiment.

FIG. 16 is a dictionary matching program 1 in the embodiment.
52 is a flowchart illustrating an example of the flow of processing after startup.

FIG. 17 is an object candidate enlarging program 1 in the embodiment.
53 is a flowchart illustrating an example of the flow of processing after activation.

FIG. 18 is a flowchart illustrating an example of a flow of processing after activation of a display object knowledge table creation program 18 in the embodiment.

FIG. 19 is an explanatory diagram of a buffer memory used in the embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Information processing device, 2 ... Main storage device, 3 ... Panel control device, 4 ... Display, 5 ... Touch panel, 6 ... Display control device, 7 ... A / D converter, 8 ... Microphone, 11 ... System program, 12 ... a figure editing program, 13 ... a sound recognition program, 14 ... a pointing coordinate reading program, 15 ... an object instruction program, 16 ... sound standard pattern data, 17 ... a word dictionary, 18 ... a display object knowledge table creation program,

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-4-372012 (JP, A) JP-A-60-146327 (JP, A) JP-A-59-183429 (JP, A) 178742 (JP, U) (58) Field surveyed (Int. Cl. ⁷ , DB name) G06T 11/80 G06F 3/16 320 G06F 3/03 380

Claims (8)

(57) [Claims] An object instruction in a combined form, in which when pointing one of a plurality of objects displayed on a screen of a display device as an object, a pointing instruction by a finger or a pen and an instruction by voice are used together. A method, wherein a mutual positional relationship between the plurality of display objects is obtained, a display object knowledge table representing the positional relationship is created, and a voice recognition result for an instruction of the display object by voice including the positional relationship is obtained. A candidate for the object indicated by the voice is obtained based on the contents of the display object knowledge table, position information based on the pointing instruction is obtained, and an object is specified based on the position information based on the pointing instruction and the object candidate. A method of pointing an object in a composite form, comprising: 2. The method according to claim 1, wherein, among the object candidates having the highest likelihood based on the voice recognition, an object candidate located at a position closest to the position indicated by the pointing instruction is specified as the object. An object pointing method using a composite form. 3. An instruction range is obtained from position information based on the pointing instruction, and when an object candidate with the highest likelihood by the voice recognition exists in the instruction range, the object candidate is set as an object. The method according to claim 1, wherein the object is specified. 4. When there are a plurality of object candidates having the highest likelihood in the indicated range, an image area including the plurality of object candidates is enlarged, and thereafter, an instruction and a voice using the finger or the pen are again performed. The method according to claim 3, wherein the display object is instructed by a multi-function device. 5. When there is no object candidate having the highest likelihood in the indicated range, and when an object candidate having the next highest likelihood exists in the indicated range, the object candidate is set as the object. 5. The method according to claim 4, wherein the object is specified. 6. The method according to claim 3, wherein the pointing range is a predetermined range near the pointing coordinates of the finger or the pen. 7. An information display means for displaying information, a pointing means for indicating a position on the display screen of the information display means using a finger or a pen, a voice recognition means for recognizing a voice instruction. A composite form input device that uses one of a plurality of display objects on the display screen as an object using both a pointing instruction and a voice instruction, wherein at least a plurality of display objects are displayed on the display screen. Means for storing a word dictionary including a word representing the name of the displayed object and a word representing the mutual positional relationship of the plurality of displayed objects; means for determining the mutual positional relationship between the plurality of displayed objects; Means for storing a display object knowledge table indicating the position information; means for obtaining position information based on the pointing instruction; and a name and a name of the display object recognized by the voice recognition means. And a target specifying means for determining a target object from the candidate object by comparing the positional relationship with the display object knowledge table to obtain a target object candidate and referring to the position information. Shape input device. 8. An enlarging means for enlarging the object candidates while maintaining a positional relationship when one object cannot be identified from a plurality of object candidates by the object specifying means. Item 7. The composite form input device according to Item 7.