JPH09114634A - Multi-modal information integrated analysis device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To integrally analyze voice sounded by a human being and the gesture of the human being and to output an analyzed result. SOLUTION: A voice recognition part 11 voice-recognizes sounded voice and outputs a voice recognized result and the time corresponding to it and a language analysis part 12 performs language analysis by using knowledge relating to languages and outputs the semantic structure of the voice recognized result and the corresponding time. A GUI control part 13 outputs the position on a screen of the track of an inputted gesture and the corresponding time and a gesture analysis part 14 performs analysis by using the knowledge relating to a diagram provided with plural instruction object candidates for information from the GUI control part 13 and outputs the kind of the gesture, the time corresponding to it and an instruction object which is the instruction object candidate instructed by the gesture. An integrated analysis part 15 detects the timewise relation of a retrieved word or phrase corresponding to the gesture and the instruction object and generates the semantic structure for which the semantic structure of the voice recognized result and the semantic structure of the kind of the gesture are integrated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multi-modal information integrated analysis apparatus for performing integrated analysis of an input human speech and an input human gesture and outputting an analysis result.

[0002]

2. Description of the Related Art FIG. 16 shows a prior art document "Tsunio Nitta et al.," A study of a multi-modal dialogue system using free-speech speech input and direct finger (direct instruction) ", Institute of Electronics, Information and Communication Engineers technical report. , SP92-120, 1993 1
2 shows a conventional multi-modal dialogue system disclosed in "Moon". In this document, both the input means (free speech voice + direct finger (touch)) and the output means (response sentence voice synthesis + graphics) are made into multiple channels, and
While monitoring the user's situation with multiple sensors, we made a prototype of a real-time dialogue system that gives instructions, guidance, and screen development according to the "plan" for "prospect", and described the results of studying its application to the information guidance system. Has been.

As shown in FIG. 16, this conventional multi-modal dialogue system includes a user situation detecting section 101,
The word spotter 102, the input management unit 103, the “probability” creation unit 104 for the user's action, the plan creation unit 105, the response strategy creation unit 106, and the sentence-speech conversion unit 1
07 and a task model 110 stored in the memory, and the user situation detection unit 101 has three sensors S1 and S1.
2, S3 are connected, the touch panel T1 is connected to the input management unit 103, and the word spotter 102 is connected.
A CRT display C1 is connected to the response strategy generation unit 106, and an external speaker H1 and a handset H2 are connected to the sentence-speech conversion unit 107.

Next, the operation of the conventional multimodal dialogue system configured as described above will be described below. (1) When the user comes in front of the system, the sensor S1 detects it. Next, the system is the CRT display C
The message “Please put the handset on your ear.” And the screen of the handset H2 that describes how to hold the handset are displayed in 1. At the same time, the instruction "Please put the handset on your ear." Is output from the external speaker H1 as a synthetic sound. (2) Next, when the handset H2 is picked up, the sensor S2 detects it and displays the "voice guidance screen". (3) Then, when the handset H2 is applied to the ear, the sensor S3 detects it and displays the instruction "Speak your desired location." At the same time, the voice output is switched from the external speaker H1 to the speaker built in the handset H2, and the same content is guided by voice. (4) When you say a word to be guided, for example, "I want to go to a department store?", Multiple department stores around Tokyo Station are displayed on the map. (5) If you touch the display of each department store name with your finger, the information of that department store (“Today is a regular holiday” etc.) can be obtained by synthesized sound.

[0005]

In the above-described conventional multimodal dialogue system, the word spotting voice recognition device for recognizing only the synthesized voice is used in the voice input unit, and the language analysis device is not used. ,
A detailed interpretation of the overall meaning of the user's utterance is not possible. On the other hand, regarding gestures, only the operation of touching and selecting graphics on the screen is accepted. That is, although the item information can be selectively recognized, it is not possible to interpret the meaning of a complicated gesture performed by a human such as "encircling". Furthermore, in the conventional example, there is a problem in that it is not possible to analyze the integrated meaning of a combination of various utterances and complicated gestures, which is used in a real city.

An object of the present invention is to solve the above problems and provide a multi-modal information integrated analysis device capable of comprehensively analyzing a human uttered voice and a human gesture and outputting an analysis result. To do.

[0007]

According to a first aspect of the present invention, there is provided a multimodal information integrated analysis apparatus which outputs time information from a predetermined reference time and time information output from the time measurement means. A voice recognition means for voice-recognizing the input speech voice based on the time information and outputting the voice recognition result together with the time information corresponding to the voice recognition result, and a voice recognition result output from the voice recognition means. Language analysis means for performing language analysis using knowledge about a predetermined language based on the time information corresponding to the time information, and outputting the semantic structure of the speech recognition result together with the time information corresponding thereto, and a plurality of pointing objects. Based on the time information output from the input means for displaying a figure including candidates on the screen and inputting a human gesture on the displayed screen and the time measuring means. With respect to the position on the screen of the trajectory of the gesture input through the input unit, the interface control unit that outputs together with the corresponding time information, and the position on the screen of the trajectory of the gesture output from the interface control unit. Then, by analyzing using the knowledge about the diagram including the plurality of pointer candidates, the type of the gesture, time information corresponding to it, and the pointer designated by the gesture among the plurality of pointer candidates. Gesture analysis means for outputting the information of the candidate referent, the semantic structure of the speech recognition result output from the language analysis means and time information corresponding thereto, and the gesture output from the gesture analysis means. Based on the type, time information corresponding to it, and the information on the pointing object, the voice recognition result The word or phrase corresponding to the gesture is searched from the taste structure, the word or phrase corresponding to the searched gesture is detected, and the temporal relationship between the information of the pointing object is detected, and based on the detected temporal relationship. In addition, an integrated analysis unit that generates and outputs a semantic structure in which the semantic structure of the voice recognition result and the semantic structure of the gesture type are integrated is provided.

A multimodal information integrated analysis apparatus according to a second aspect is the multimodal information integrated analysis apparatus according to the first aspect, characterized in that the word corresponding to the gesture is a demonstrative. Furthermore, the multimodal information integrated analysis device according to claim 3 is the multimodal information integrated analysis device according to claim 1 or 2, wherein the type of the gesture analyzed by the gesture analysis means is a "circle" gesture. , "Draw a line" gesture, "Dot" gesture, "Marking"
It is characterized by including a gesture and a "scrambling" gesture, which is a depiction with random movement.

Furthermore, the multimodal information integrated analysis device according to claim 4 is the multimodal information integrated analysis device according to claim 1, 2 or 3, wherein the gesture analysis means is a rectangle surrounding the trajectory of the gesture. It is characterized in that the rectangle is divided into a plurality of regions by a plurality of lines passing through the center, and the type of the gesture is determined based on the relationship between the divided regions and the trajectory of the gesture.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a multi-modal information integrated analysis device according to an embodiment of the present invention. The multimodal information integrated analysis device of this embodiment is, for example, a route guidance system using a map, in which a map near Kyoto station is displayed on the CRT display 33, and the user says “Kyoto station is here through the microphone 31. At the same time as saying "", the explanation will be made assuming that the user's finger is used to perform a pointing gesture in which Kyoto Station is circled, for example. Here, the multi-modal information integrated analysis device uses as input the information of the gesture made by drawing a line or the like on the figure or the picture on the screen of the CRT display 33 and the information of the speech voice made at the same time. Receive and map database 24
The knowledge data about the display object on the screen of the CRT display 33 stored in advance and the hidden Markov network (hereinafter, referred to as HM network) 2 stored in each memory 2 respectively.
1, the context-free grammar 22, and the semantic structure of the uttered voice and the semantic structure of the gesture input using the knowledge of the language such as the word dictionary 23, respectively, and then, the semantic structure of the uttered voice and the semantic structure of the gesture are analyzed. The integrated analysis is performed as time passes and the analysis result is output.

As shown in FIG. 1, the multimodal information integrated analysis apparatus of this embodiment has a supervisory control unit 10, a voice recognition unit 11, a language analysis unit 12, and a graphic user interface control as processing units for executing various control processes. Unit (hereinafter, referred to as GUI control unit) 13, gesture analysis unit 14, and integrated analysis unit 15 (hereinafter collectively referred to as processing unit 10).
It is called -15. ). Here, each processing unit 11-
The voice control unit 10, the language analysis unit 12, the GUI control unit 13, the gesture analysis unit 14, and the integrated analysis unit 15 are connected to the monitoring control unit 10 that comprehensively monitors and controls 15. Also,
A clock signal generated by a clock signal generator 30 whose set / reset is controlled by the supervisory control unit 10 is input to each processing unit 11-14.
In -14, the time when the start button 32a is turned on, that is, the clock time from the onset time (in the present embodiment, expressed in units of milliseconds) is calculated based on the clock signal. It becomes the reference time of the multimodal information of the device.

As input information devices from the user, a microphone 31, a keyboard 32 having a start button 32a, a stop button 32b, and a quit button 32c, a CRT display 33 having a touch panel on the screen, and a mouse 34 are provided. , Where the microphone 31 is connected to the voice recognition unit 11 and the keyboard 3
2 is connected to the monitoring control unit 10 and a CRT display 3
3 and the mouse 34 are connected to the GUI control unit 13. On the other hand, a CRT display 35 is provided as an output information device and is connected to the integrated analysis unit 15.

An HM network 21 and a context-free grammar 22 are connected to the voice recognition unit 11, and a context-free grammar 22 and a word dictionary are connected to the language analysis unit 12. The gesture analysis unit 14 is connected to the map database 24 and the gesture dictionary 25, and the integrated analysis unit 15 is connected to the gesture dictionary 25.

Here, the monitor control unit 10 and the voice recognition unit 11
The language analysis unit 12, the GUI control unit 13, the gesture analysis unit 14, and the integrated analysis unit 15 are each configured by, for example, a digital electronic computer, and each processing unit 10-15 is
It has a CPU, a ROM that stores an operation program and data for executing the operation program, and a RAM that is used as a working memory. In addition, the six processing units 10-15
May be composed of one digital computer. Furthermore, the HM network 21, the context-free grammar 22, the word dictionary 23, the map database 24, and the gesture dictionary 25 are stored in a memory such as a hard disk memory.

First, each database connected to each processing unit 11-15 will be described below. The HM network 21, the context-free grammar 22 and the word dictionary 23 are databases of language knowledge for speech recognition and language analysis,
The map database 24 is a database of knowledge about display objects or pointing objects displayed on the screen of the CRT display 33, and the gesture dictionary 25 is on the screen of the CRT display 33 (the screen operates as a so-called touch panel) by the user. It is a database of knowledge for identifying the type of gesture performed in.

In the word dictionary 23, there are 43 words related to the route guidance task in the word dictionary 23,
A plurality of words and their attributes are represented by feature structures and are augmented with time information for capturing acoustic information and spatial information of gestures. Table 1 shows an example of the word dictionary 23.

[0017]

[Table 1] Feature structure of direct expression in word dictionary 23 ─────────────────────────────────── ( deflex-named around this-1 around this n-deictic! (lex-phon-orth "konoatari" "this around") (<! m sem> == [[RELN DEITIC-PLACE] [AGEN * SPEAKER *] [RECP * HEARER *] [OBJE [[RELN this area]]]]) (<! M time-stamp> == [[SPEECH [[tS? X1] [tE? X2]]]])) (<! M gesture> == [[RELN CIRCLING-3] [LOCATION [[lS [[X? X] [Y? Y]]] [lE [[X? X] [Y? Y]]]]] [TIME-STAMP [[ mouse [[tS? X1] [tE? X2]]]]])) (<! m prag> == [[iterr agen]])) ──────────────── ────────────────────

The contents of Table 1 will be described below.
The first line is the definition of the directive "this area" in the dictionary for language analysis, and the first line "(deflex-named this area-1 this area n-deictic" is "this area". "
It has an index of and has a part of speech called n-deictic, which defines the word "this area -1". "! (Lex-phon-orth" konoatari "" this area ")" on the second line
The sound and the notation mean "konoatari" and "this area", respectively. 3rd to 6th line "

(Equation 1) Defines the relationship name of DEITIC-PLACE, the subject of * SPEAKER *, the receptor of * HEARER *, and the object of "this area" as the attribute of meaning (sem).
The 7th line "(<! M time-stamp> == [[SPEECH [[tS? X1]
[tE? X2]]]]) ”defines the start time and end time of the utterance as attributes of time information (time-stamp). 8th
From line 12 to line 12

[Equation 2] (<! M gesture> == [[RELN CIRCLING-3] [LOCATION [[lS [[X? X] [Y? Y]]] [lE [[X? X] [Y? Y] ]]]] [TIME-STAMP [[mouse [[tS? X1] [tE? X2]]]]]]]] is C as an attribute called gesture information (gesture).
It defines the relationship name IRCLING-3, the location on the display where the gesture was made, the means by which the gesture was made, and its start and end times. Line 13 "(<! M prag>
== [[iterr agen]])) ”defines the holder of the information as a pragmatic attribute.

In the gesture dictionary 25 of this embodiment, there are only eight entries (headwords). The feature in the feature structure of the gesture is configured to capture the temporal and spatial information of the gesture. Table 2 shows an example of the gesture dictionary 25.

[0020]

[Table 2] Gesture feature structure in gesture dictionary 25 ─────────────────────────────────── (deflex- named CIRCLING-3 CIRCLING gesture (<! m sem> == [[RELN CIRCLING-3] [LOCATION [[lS [[X? X] [Y? Y]]] [lE [[X? X] [Y? Y]]]]] [TIME-STAMP [[mouse [[tS? X1] [tE? X2]]]]]]) ───────────────────── ───────────────

Explaining Table 2, "(defle
x-named CIRCLING-3 CIRCLING gesture "is CIRCLIN
It defines a gesture called CIRCLING-3 (“enclose circle” -3), which has an index of G and a part of speech of gesture. From the 2nd line to the 6th line

(Equation 3) Is the same as that in Table 1. As attributes of gesture information (gesture), the relationship name of CIRCLING-3, the place where the gesture is performed on the display, the means by which the gesture is performed, and the start / end time thereof are given. Define.

Next, in the map database 24, the object or pointer candidate on the map is represented with a list of attributes. Table 3 shows an example of the map database 24.

[0023]

[Table 3] Map representation in the map database 24 ─────────────────────────────────── [Object number] [min X] [min Y] [max X] [max Y] [kind of object] [name of object] example: [1] [56] [145] [70] [178] [hotel] [kyoto-hotel ] ───────────────────────────────────

The explanation of Table 3 shows that the candidate number for the indicator is 1.
And the minimum x coordinate value (min x) on the screen of the CRT display 33 of the pointer candidate is 56, and the minimum y coordinate value (min y) of the pointer candidate.
Is 145, the maximum x coordinate value (max x) of the pointer candidate is 70, and the maximum y coordinate value (max y) of the pointer candidate is 178. In addition, the type of the indicator candidate is “hotel”, and the name of the indicator candidate is “Kyoto Hotel”.

HM network 21 connected to the voice recognition unit 11
Is represented as a plurality of networks with each state as a node, using the efficient representation format of the phoneme environment-dependent hidden Markov model, and each state has the following information. (A) State number (b) Acceptable context class (c) List of preceding states and succeeding states (d) Parameter of output probability density distribution (e) Probability of self-transition and transition to subsequent state The grammar 22 contains 114 grammar rules used for both speech recognition and linguistic analysis. The number of terms is 43 words and has a phoneme perplexity of 1.74.

The monitoring control unit 10 includes all the processing units 11-
It is a multi-function module that controls 15 and controls the data flow. FIG. 2 is a monitoring control process executed by the monitoring control unit 10, which will be described below with reference to FIG.

First, in step S1, the start button 32
It is determined whether or not a has been turned on. When it is turned on, the process proceeds to step S2, and when it is not turned on, step S1 is performed.
Is repeated. In step S2, all processing units 11-15 are initialized. At this point, the voice recognition unit 11 starts detection of voice input from the microphone 31, executes voice recognition processing, and outputs the voice recognition result to the monitoring control unit 10. On the other hand, the GUI control unit 13 starts detecting data regarding a gesture input by the user on the screen of the CRT display 33 whose screen is a touch panel and data regarding a gesture input by the user operating the mouse 34, The data regarding the detected gesture (specifically, the coordinate value of each point of the trajectory of the gesture on the screen) is output to the monitoring control unit 10. At the same time, in step S3, the clock signal generator 3
After resetting 0 and resetting the system clock time (hereinafter referred to as clock time) generated by the clock signal generator 30 to 0, the clocking is started. The system clock time is from the clock signal generator 30 to each processing unit 1.
It is output to 1-14. As a result, a turn, which is one processing period, is started.

In step S4, the data transfer process is executed, and the monitor control unit 10 transfers the data of the voice recognition result with the clock time information output from the voice recognition unit 11 to the language analysis unit 12 and the GUI control. The data regarding the gesture with the clock time information output from the unit 13 is transferred to the gesture analysis unit 14. At this time,
The language analysis unit 12 executes a language analysis process as described later based on the data of the voice recognition result with the clock time information, and monitors the data including the semantic structure of the voice with the clock time information, which is the processing result. Output to 10. On the other hand, the gesture analysis unit 14 analyzes the type of gesture as described later based on the data about the gesture with the clock time information, and monitors and controls the data including the semantic structure of the gesture with the clock time information, which is the processing result. Output to the unit 10.

Next, in step S5, it is determined whether or not the stop button 32b is turned on. If it is not turned on, it means that the turn is still in progress. Therefore, the process of step S5 is repeated. S
In step 6, the processing units 11-14 are notified of the end of the turn. Then, in step S7, the clocking of the clock signal generator 30 is stopped. In step S8, the data transfer process is executed, and the monitor control unit 10 detects the data including the semantic structure of the voice with the clock time information from the language analysis unit 12 and the gesture with the clock time information from the gesture analysis unit 14. The data including the semantic structure is output to the integrated analysis unit 15. Then, in step S9, the integrated analysis unit 15 is caused to execute an integrated analysis process, which will be described later, and the analysis result is output and displayed on the CRT display 35. Next, in step S10, it is determined whether or not the quiz button 32c is turned on. If it is turned on, the monitoring control process is ended, and if it is not turned on, the process returns to step S1 to repeat the above process. .

One of the most important processes of the monitor control unit 10 is "event collection" in step S8, that is, one turn ("onset time" and "offset time" as shown in FIG. All of the peripheral events (voice, gesture, etc.) that occur in (between “and”) are collected and passed to the integrated analysis unit 15. For the data of the semantic structure of the voice, the start time and the end time are given for each word, and for the data of the semantic structure of the gesture,
A start time and an end time are given to each gesture. Here, the onset time is the start time of one turn and is the time when the user turns on the start button 32a. The offset time is the end time of the turn and is the time when the user turns on the stop button 32b.

The speech recognition unit 11 is a phoneme synchronization type SS which has been researched and developed at the ATR Speech Translation and Communication Research Institute.
S-LR technology (see, for example, the prior art document Harald
Singer et al. , "A Modular
Speech Recognition System
m Architecture ”, Proceedin
gs of Acoustic Society, Ja
pan, Fall, 1994 ”. ) Based on the prior art continuous speech recognizer, which was developed with emphasis on the modular system so that new modules can be easily added.

The voice uttered by the user is input to the microphone 31 and converted into a voice signal, and then feature extraction processing is executed. In this process, after A / D converting the audio signal,
For example, LPC analysis is performed to extract 34-dimensional characteristic parameters including logarithmic power, 16th-order cepstrum coefficient, Δ logarithmic power, and 16th-order Δcepstrum coefficient. Next, the phoneme matching process and the LR parser process are executed on the extracted feature parameter. Here, the likelihood for the data in the phoneme matching section is calculated using the HM network 21 which is an unspecified speaker model, and the value of this likelihood is returned to the LR parser as a phoneme matching score, and the LR parser is context-free. The phoneme prediction data input with reference to the grammar 22 is processed from left to right without any backtracking. If there is syntactic ambiguity, the stack is split and the analysis of all candidates is processed in parallel. The LR parser predicts the next phoneme based on the context-free grammar 22 and outputs phoneme prediction data. In the phoneme matching process, the HM corresponding to the phoneme is calculated.
The information of the network 21 is referred to, the likelihood is returned to the LR parser as a speech recognition score, and the phonemes are sequentially connected to perform continuous speech recognition. Information is output to the monitoring control unit 10. When a plurality of phonemes are predicted in the above continuous speech recognition processing, the existence of all of them is checked, and the pruning that leaves a partial tree with a high likelihood of partial speech recognition by the beam search method. To achieve high-speed processing.

The plurality of sentences recognized in the speech recognition unit 11 of this embodiment are those most short and simple, they are, for example, straight-called "Kyoto hotel to <br/> around this" shows Includes examples of typical expressions. Multiple sentences can be spoken in either continuous mode or concatenated mode. The user can speak one sentence freely in one breath, or one pose can be freely placed between phrases of two clauses. The output from the voice recognition unit 11 is, for each word, three elements of the recognized word, start time, and end time. Table 4 shows an example of the voice recognition result output from the voice recognition unit 11.

[0034]

[Table 4] Speech recognition results ─────────────────────────────────── sentence: Kyoto Hotel is around here Or recognition results: 1135: time elapsed since the turn "onset time" Kyoto Hotel 0 830: speech onset & offset time is 830 920 3842: time elapsed since the turn "onset time" 0 780: speech onset time reset due to 780 860 or 860 1050 or 1050 1200 at the pause Kyoto Hotel is around here -32.115994 ─────────────────────────────── ─────

Explaining Table 4, the meanings of the contents of Table 4 are as follows. The sentence of the voice recognition result is "Is this the Kyoto Hotel?", And the start time of the "Kyoto Hotel" in this sentence (hereinafter referred to as the first start time) is 1135 mm from the onset time. This is when the elapsed time of seconds has elapsed. Here, “Kyoto Hotel” in the sentence of the voice recognition result is 0 based on the first start time.
It is in the time from millisecond to 830 milliseconds, and "ha" is 8
It is in the time from 30 ms to 920 ms. The start time of "about this" in the above sentence (hereinafter referred to as the second start time) is when the elapsed time of 3842 milliseconds has elapsed from the onset time. Here, “this area” is 0 to 780 seconds based on the second start time.
It is in the time to millisecond, and "de" is from 780 millisecond to 8
The time is up to 60 ms, the "su" is from 860 ms to 1050 ms, and the "ka" is 1050
It is in the time from millisecond to 1200 milliseconds. And
The score of the sentence "Is this the Kyoto Hotel?" In the voice recognition result is "-32.115994".

The language analysis unit 12 performs parsing (sentence analysis).
Toolkit (see prior art document "Toshihisa"
Tashiro et al. , "A Parsing
Toolkit for Spoken Langu
age Processing ”, WGNL Meet
ing of IPSJ, 95-NLP-106, 1
995 ". ), This parsing toolkit is developed with emphasis on efficient unification and modularity to handle the phonemes of many words in instant speech. The input data to the language analysis unit 12 is the result of voice recognition. When the speech recognition result is received, the language analysis unit 12 first generates a parse tree using the grammar rules in the context-free grammar 22, then transforms the tree into a dependency structure, and finally, the utterance of the utterance. Data including a semantic feature structure (Table 5), that is, a semantic structure of voice and clock time information is generated. Then, the data is handed to the integrated analysis unit 15 via the monitoring control unit 10. Here, an example of the output result of the language analysis unit 12 is shown in Table 5.

[0037]

[Table 5] Output results of the language analysis unit 12 ─────────────────────────────────── sentence: Kyoto Hotel Is this around [SEM [[RELN * YN-QUESTION *] [AGEN * SPEAKER *] [RECP * HEARER *] [OBJE [[RELN * BE-LOCATED *] [IDEN [[RELN * Kyoto Hotel *]] ] [PLACE [[RELN * DEICTIC-PLACE *] [AGEN * SPEAKER *] [RECP * SPEAKER *] [OBJE [[RELN * this area *] [PRAG [[ITERR * SPEAKER *]]] [TIME-STAMP [ [SPEECH [[tS? X1] [tE? X2]]]]] [GESTURE [[RELN CIRCLING-3] [LOCATION [[lS [[X? X] [Y? Y]]] [lE [[X? X] [Y? Y]]]]]] [TIME-STAMP [[mouse [[tS? X1] [tE? X2]]]]]]]]]]]]]]] ──────── ────────────────────────────

Explaining Table 5, in the first line, the sentence of the input character string is "Is this the Kyoto Hotel?", And in the second line, "[SEM [[RELN * YN- QUESTION
*] ”Means that the utterance intent of this sentence is * YN-QUESTION * (question answered with yes or no). 3rd and 4th line "

[Equation 4] [AGEN * SPEAKER *] [RECP * HEARER *] ”is the same as in Table 1, the main body of * SPEAKER *, * HEARER *
Defines the receptor. And, in the 5th line, "[OBJE
[[RELN * BE-LOCATED *]], the question content is "thing (IDEN)
Is in a place (PLACE) ”. "[IDEN [[RELN * Kyoto Hotel *]]]" on the 6th line is
"The thing is" Kyoto Hotel ". It means that. Furthermore, from the 7th line to the 17th line,

[Equation 5] [PLACE [[RELN * DEICTIC-PLACE *] [AGEN * SPEAKER *] [RECP * SPEAKER *] [OBJE [[RELN * this area *] [PRAG [[ITERR * SPEAKER *]]] [TIME -STAMP [[SPEECH [[tS? X1] [tE? X2]]]]] [GESTURE [[RELN CIRCLING-3] [LOCATION [[lS [[X? X] [Y? Y]]] [lE [ [X? X] [Y? Y]]]]] [TIME-STAMP [[mouse [[tS? X1] [tE? X2]]]]]]]]]]]]]]]] Indicates that this is what is indicated by "this area".

The GUI control unit 13 manages the user interface by displaying, for example, the graphics screen shown in FIG. 6, and detects a plurality of screen events (for example, a plurality of gestures on the touch panel) on the screen of the CRT display 33. To monitor. In particular, the GUI control unit 13 executes the following processing. (A) As shown in the lower part of FIG. 6, a map including a plurality of indicator candidates (buildings, stations, etc.) and other graphics are displayed. (B) Read out the coordinate values corresponding to the trajectory of the gesture on the map by the user. (C) Push buttons 32a, 32 of the keyboard 32
A plurality of events b and 32c are detected. (D) As shown in the upper part of FIG. 6, the integrated analysis result executed by the integrated analysis unit 15 (this is input from the integrated analysis unit 15 to the GUI control unit 13 via the monitoring control unit 10). The result of temporal matching between the result and the gesture analysis result is displayed. It also displays a unified semantic representation of utterances and gestures.

The main processing of the gesture analysis unit 14 is as follows. 1) Recognizing the types of indirect gestures (“circle”, “draw line”, etc.), 2) selecting a candidate for an indicator (object), and 3) the time of the gesture. And spatial information (eg Table 6
Shown in ) Is generated. As shown in FIG. 3, the gesture analysis process executed by the gesture analysis unit 14 includes a gesture recognition process in step S11 and a pointing object selection process in step S12.

In the gesture recognition process of step S11, the following process is executed. First, the points of the entire locus of one gesture (x, y coordinate values on the screen of the CRT display 33) input from the GUI control unit 13 via the monitoring control unit 10 are stored in the memory. Then
As shown in FIG. 5, the minimum values (min x) and (min y) and the maximum values (max x) and (max y) of the x and y coordinate values of the stored locus points are calculated and the center thereof is calculated. Find the O point. Then, as shown in FIG. 5, for example, in the case of the “circle” gesture 600, the minimum x and y coordinate values (min x) and (min
y) and the region of the “circle” gesture 600 located within the rectangle of maximum values (max x) and (max y) is divided into eight regions A1 to A8, and each region A1 to A8
The coordinate value belonging to 8 is calculated.

Then, if the coordinate values of the points of the trajectory of the gesture 600 exist in all areas A1 to A8, the start point 601 and the end point 6 of the gesture 600 are displayed.
If the Euclidean distance to 02 is smaller than 50 (current allocation design value), it is determined that the gesture 600 is “encircled”. Also, if there are multiple trajectory points in only one area, then the gesture is "pointing (pointing, pointing,
Or hit a dot) ”. If area A6
, And when there is no locus point at A7 and the Euclidean distance between the start point 601 and the end point 602 of the gesture 600 is smaller than 3 (current assigned design value), the gesture is “marking It is determined to be. When the remaining conditions are satisfied, it is determined that the gesture is a “draw line” gesture.

Next, in the pointing object selection processing in step S12, the processing is executed as follows. here,
The pointing candidate is a building or station on the map of the CRT display 33, such as Kyoto Hotel or Kyoto Station. (A) When it is determined that the gesture is a “circle encircle” gesture, the indicator close to the center is selected from all the indicator candidates in or around the circle. (B) When it is determined that the gesture is the "pointing" gesture, the pointing object positioned at the pointing target is selected. (C) When it is determined that the gesture is “draw a line”, the pointing object located on the locus is selected. (D) When it is judged as a "marking" gesture,
The indicator closest to the center is selected. Table 6 shows the temporal and spatial information of the gesture, which is the analysis result output from the gesture analysis unit 14 (or,
It is called the semantic structure of gestures. ).

[0044]

[Table 6] Temporal and spatial information of gestures ─────────────────────────────────── 3: turn ID circle: gesture analysis result 3119: gesture onset time 4864: gesture offset time (897,921) (128,164): object coordinates ───────────────────────── ──────────

Referring to Table 6, the first line shows the turn ID number, and the second line shows that the gesture of "encircling" is determined. The third line means that the time of the start point of the gesture is 3119 milliseconds counted from the onset time, and the fourth line is that the time of the end point of the gesture is the onset time. Means that it is 4864 milliseconds. And
The fifth line means the coordinate value of the pointing object designated by the gesture, specifically, a set of (max x, maxy) and (min x, min y).

That is, the gesture analysis unit 14 recognizes the type of gesture from the coordinate information, and the result and the CRT.
From the knowledge of the drawings and pictures on the screen of the display 33, the pointing object indicated by the gesture is estimated and judged. In the case of the gestures of FIGS. 5 and 6, it is possible to determine that “Kyoto Hotel” is instructed from the positional relationship between the “maru” of the gesture and the map on the screen. Finally, a semantic structure having information about the type of gesture, the time, and the referent is generated.

The integrated analysis unit 15 executes the following processing. (A) Semantic feature structure of uttered voice input from the language analysis unit 12 via the monitoring control unit 10, and the temporal and temporal structure of the gesture input from the gesture analysis unit 14 via the monitoring control unit 10. Receives spatial information. (B) A direct feature (for example, the demonstrative "here") in the semantic feature structure of the uttered voice is searched. (C) Check the temporal arrangement relationship between the explicit feature and the gesture. For example, as shown in FIG.
What is the temporal arrangement relationship between the “here” utterance and the “circle” gesture? Specifically, for example, the time of the “circle” gesture is included in the “here” utterance time. There? , If included, it is determined that there is a direct instructional relationship. Further, even if the utterance "here" is made immediately after the "enclose" gesture, it is determined that there is a direct instructional relationship. (D) The direct analysis feature structure having the temporal and spatial values of a plurality of gestures, which is the integrated analysis result, is displayed on the CRT display 35 or 33. Table 7 shows an example thereof. It should be noted that the arrangement of the direct gesture and the gesture is adjusted from the beginning of the utterance and the gesture, one gesture is assigned to one direct gesture, and the remaining gestures are ignored.

[0048]

[Table 7] Semantic Expression of Utterance Generated by Integrated Analysis Unit ─────────────────────────────────── ─ [SEM [[RELN * YN-QUESTION *] [AGEN * SPEAKER *] [RECP * HEARER *] [OBJE [[RELN * BE-LOCATED *] [IDEN [[RELN * Kyoto Hotel *]]] [PLACE [ [RELN * DEICTIC-PLACE *] [AGEN * SPEAKER *] [RECP * SPEAKER *] [OBJE [[RELN * this *] [PRAG [[ITERR * SPEAKER *]]] [TIME-STAMP [[SPEECH [[ tS 3842] [tE 4622]]]]] [GESTURE [[RELN CIRCLING-3] [LOCATION [[lS [[X 897] [Y 921]]] [lE [[X 128] [Y 164]]]] ] [TIME-STAMP [[mouse [[tS 3119] [tE 4864]]]]]]]]]]]]]]] ──────────────────── ────────────────

Explaining Table 7, the contents are as shown in Table 5.
The specific value is entered in the part. That is,
The integrated analysis unit 15 receives the semantic structure of the voice and the semantic structure of the gesture, and based on the time information of the voice and the gesture and what the gesture indicates, the portion corresponding to the gesture from the semantic structure of the voice ( The verb "here") is searched for, the semantic structure of the gesture is added thereto, and finally the semantic structure in which the meanings of the voice and the gesture are integrated is generated and output. In summary, the integrated analysis unit 15 searches for a descriptive word from the semantic structure of the speech recognition result, detects the temporal relationship between the searched denotative word and the information of the pointing object, and detects the detected temporal Based on the relationship, a semantic structure in which the semantic structure of the voice recognition result and the semantic structure of the gesture type are integrated is generated and output.

Further, a modified example of the embodiment according to the present invention will be described below.

<First Modification> FIG. 7 is a flowchart showing a gesture analysis process of a modification executed by the gesture analysis unit 14 of FIG. The gesture analysis process of this modification is roughly divided into a process for recognizing the gesture in steps S21 to S27 and a pointing object selection process in step S28. Here, step S2 of determining the type of gesture
When the gesture type can be first determined in 2-S23 and S26-S28, although not shown in FIG. 7, the control flow advances to step S29 at the time of determination.

In FIG. 7, in step S21, first,
Prefix processing is executed. Here, the x, y coordinate values (hereinafter, referred to as gesture points) of all the loci of one gesture input from the GUI control unit 13 via the monitoring control unit 10 are stored in the memory of the gesture analysis unit 14. . If there are multiple gesture points at the same x, y coordinate value, only one gesture point is stored and the rest are discarded.

Next, in step S22, a pointing determination process is executed. That is, in this process, first, as shown in FIG. 8, each minimum value (min x, miny) and each maximum value of the x and y coordinate values are calculated based on the x and y coordinate values of the locus stored in the memory. Value (max
x, max y) is calculated, and one rectangle (hereinafter referred to as the minimum rectangle) 500 that surrounds all the gesture points is calculated.
Is drawn virtually. Then, the density ratio DR defined by the following equation
Is calculated.

[0054]

DR = {(number of gesture points) / (area of the smallest rectangle 500)} × 100

Here, the area is calculated in units of predetermined values of x and y coordinates. At this time, if the density ratio DR is 90% or more, the input gesture is a gesture indicating a pointer candidate, that is,
It is determined to be a "pointing gesture". If the number of gesture points is less than 5 and the density ratio is 10% or more, the input gesture is determined to be a “pointing gesture”.

Next, in step S23, marking judgment processing is executed. Here, as shown in FIG. 9, all the gesture points are connected, and the cosine value co of the angle θ between two adjacent connecting lines connecting the gesture points co
Calculate sθ. Here, as shown in FIG. 9, the angle θ between each two connecting lines is numbered in ascending order from the starting point. Then, the following four conditions (first to fourth conditions)
When all of the above are satisfied, the input gesture is determined as “marking”.

(A) As shown in FIGS. 10A and 10B, when the angle θ <90 ° or the angle θ> 270 °,
It is defined as a peak, and when the angle θ (shown as θp in FIGS. 10A and 10B) when the cosine value cos θ exceeds 0 exists, that is, the peak exists. And (B) As shown in FIG. 11, the maximum value (max
x) The length obtained by subtracting the minimum value (min x) of the x coordinate from the x coordinate value of the point 302 that is connected from the point 301 and intersects the side of the smallest rectangle 500 in the x axis direction is defined as x1.
From the maximum value of the coordinates (max x) to the minimum value of the x coordinates (mi
The length obtained by subtracting n x) is defined as x2. At this time, the value LR of the length ratio defined by the following equation is calculated.

[0058]

(7) LR = (x1 / x2) × 100 [%]

Then, the calculated length ratio value LR is 7
The second condition is 0% or more. (C) As shown in FIG. 12A or 12B, the position of the peak corresponds to the bottom (bottom) or top of the smallest rectangle 500 (shown by 201 and 202).
Is the third condition. (D) As shown in FIG. 13, the fourth condition is that the start point and the end point of the gesture are located in the area of the uppermost 25% area on the opposite side of the peak.

Next, in step S24, a judgment process of "drawing a line" is executed. Here, if there is one gesture that is neither a "pointing" gesture nor a "marking" gesture, and the number of gesture points is less than 3, then the input gesture is a "draw line" gesture. to decide.

Next, in step S25, an intermediate process is executed. Here, all gesture points (see FIG.
(A)) is connected as shown in FIG.
As shown in (c), a plurality of points are interpolated between each two gesture points. Here, the number of interpolated points depends on the distance between each two gesture points.

Next, in step S26, a judgment process of "encircling", "drawing a line", and "scrambling" is executed. Here, "scrambling" refers to depiction input having random movements, such as circles and lines, rather than a predetermined shape. In the process of step S26,
First, as shown in FIG. 14D, the smallest rectangle 500
A line connects the center O of each of the gesture points to each gesture point, and the connecting lines extend until they reach each side of the smallest rectangle 500. Here, the extended line is hereinafter referred to as an extended line. Next, the number of times the extended line intersects a plurality of gesture lines is counted, and the intersection is hereinafter referred to as a gesture line intersection.

If the number of gesture line crossings for each gesture point is 3 or more, it is determined that the input gesture is "scrambling". Also, when 85.5% or more of the extended lines have two gesture line intersections, it is determined that the gesture is a “circle” gesture. If less than 15% of the extended lines have one or less gesture line intersections and more than 75% of the extended lines have two gesture line intersections, the entered gesture is "circled". To judge. Further, if there are no extended lines that have no gesture line intersections and more than 40% of the extended lines have two gesture line intersections, then the input gesture is a "circle" gesture.
Still further, if 70% or more of the extended lines do not have a gesture line intersection, the input gesture is determined to be a "draw line" gesture.

Next, in step S27, the judgment process of "drawing a line" and "circling" is executed. In this process, as shown in FIG. 15, a horizontal line (a line parallel to the x-axis direction) and a vertical line (a line parallel to the y-axis direction) (hereinafter referred to as parallel lines) are intersected at each gesture point. Called a line.)
And are drawn virtually. If 70% or more of the extended lines intersect at only one place, the input gesture is determined to be a "draw line" gesture. If there are no extended lines that intersect at three or more places, or if the number of extended lines that intersect at only one place is less than 30%, the input gesture is determined to be a “circle” gesture.

Further, in step S28, a judgment process of "drawing a line" is executed. In this process, when the input gesture does not meet the above conditions, it is determined to be a "draw line" gesture.

Next, in step S29, a pointing object selection process is executed. When the gesture type determination processing described above makes the following determination, the pointing object selection processing differs as follows depending on the type. And
When the pointing object is selected, the gesture analysis unit 14
Data indicating the semantic structure of the gesture is output to the integrated analysis unit 15 via the monitoring control unit 10.

(A) When it is judged that the gesture is a "circle encircling" gesture, the center O of the smallest rectangle 500 is selected among all the indicator candidates that are either inside or on the periphery of the circle.
One of the indicator candidates closest to is selected as the indicator designated by the user. (B) When it is determined that the gesture is the “pointing” gesture, the indicator candidate located in the pointing indicator candidate is selected as the pointer designated by the user. (C) When it is determined that the gesture is a "draw line" gesture, the indicator candidate located on the trajectory is selected as the indicator designated by the user. (D) When it is determined to be the “marking” gesture, the pointer candidate closest to the center O of the smallest rectangle 500 is selected as the pointer designated by the user.

<Second Modification> For example, assume that a form such as an application form is filled. Consider a case where the user says "Do you write your name here?" And at the same time marks a column in the form with your finger.
The multimodal information integrated analysis device analyzes the gesture type and determines that the gesture is a “circle” gesture, as in the above embodiment. Considering that there is a form instead of a map on the screen, finally determines that the gesture points to a specific field in the form. That is, the map database 24 of FIG. 1 replaces the form database including the form style, but the other configurations are the same as those of FIG. Then, as in the above-described embodiment, the integrated analysis unit 15 analyzes and identifies that what is instructed corresponding to “here” is one column in the form, and the analysis result is displayed on the CRT display 35.
Displayed on the screen of.

<Third Modification> CRT display 33
It is assumed that the three-dimensional object A is displayed on the screen. At that time, the user utters "Please rotate this.", And traces the screen with the finger in the direction (for example, clockwise) to rotate (for example, draw an arc in the right direction). . At this time, the gesture analysis unit 1
4 first determines that the gesture is a "draw line" gesture. Then, based on the database relating to the shape and position of the object instead of the map database 24,
Based on the positional relationship between the object A and the gesture on the screen of the CRT display 33, it is determined that the gesture-instructed object is the object A, and finally, the type of the gesture, the time, and the data of the instructed object. Hand it to the integrated analysis unit 15. From the semantic structure of the voice input from the language analysis unit 12 via the monitoring control unit 10, the integrated analysis unit 15 knows that the utterance "rotate something" is made, so the "draw a line" The "gesture" corresponds to "here" indicating the "rotate" method, and is identified and determined to mean the direction of rotation "to the right". Finally, the integrated meaning of "rotate the object A to the right." Is analyzed and identified, and the analysis result is displayed on the screen of the CRT display 35.

<Fourth Modification> CRT display 33
It is assumed that a map is displayed on the screen and the user makes a gesture to draw a line along the road on the map at the same time as the utterance "I'm going like this". In this case, the coordinate values near the start point, end point, and passing point of the line are extracted, as indicated by the "draw line" gesture. Then, the integrated analysis unit 15 associates the word “in this way” with the starting point of the extracted line, the ending point, and the coordinate values in the vicinity of the passing point, and maps “the starting point to the passing point” on the map. The integrated meaning of "go through to the end point" is analyzed and identified, and the analysis result is displayed on the screen of the CRT display 35.

As described above, according to the multimodal information integrated analysis apparatus of the present embodiment, it is possible to perform an integrated analysis of a voice uttered by a human and a gesture of the human and output the analysis result. A modal information integrated analysis device can be provided. This makes it possible to analyze and judge more complicated and concrete input information of the human based on the voice spoken by the human and the gesture of the human. Further, by applying the multimodal information integrated analysis device of the present embodiment to, for example, a voice dialogue system, it is possible to analyze the pointing gesture together with the voice as an input, and thus more flexible between the human and the system. Dialogue can be realized. Furthermore, by applying the multimodal information integrated analysis device of the present embodiment to input analysis in a multimodal translation dialogue system, for example, it is possible to translate based on a semantic structure in which utterance information and gesture information are organically integrated. Yes, and more accurate translations into other languages.

In the above-described embodiments and modified examples, the CRT display 35 is used as the output device for outputting the integrated semantic structure as the analysis result, but the present invention is not limited to this, and other image display devices are used. Other information output devices such as a printer may be provided.

In the above-described embodiments and modified examples, the values of various parameters are used in the determination of the type of gesture, but these values are design values and may be changed as necessary.

In the above-described embodiment and modification, the integrated analysis unit 15 searches the semantic structure of the voice recognition result for the indicator corresponding to the gesture, and the indicator corresponding to the searched gesture, and The temporal relationship with the information of the pointing object is detected. The present invention is not limited to this, the word or phrase corresponding to the gesture is searched from the semantic structure of the voice recognition result instead of the directive, and the word or phrase corresponding to the searched gesture and the instruction You may detect the temporal relationship with the information of a thing. Here, the word or phrase corresponding to the gesture is, for example, the following. For example, “Kyoto Hotel has vacant rooms.”
When marking "Kyoto Hotel" on the map while uttering, "Kyoto Hotel" is a word corresponding to the above gesture.

In the above embodiment, the keyboard 32 is used.
Is connected to the monitor control unit 10, but the present invention is not limited to this, and the keyboard 32 is connected to the GUI control unit 13 so that the input information input using the keyboard 32 is GU.
It may be transferred to the monitoring control unit 10 via the I control unit 13. The keyboard 32 is a CRT display 33.
It may be the keyboard of the upper touch panel.

[0076]

As described in detail above, according to the present invention, the semantic structure of the speech recognition result output from the language analyzing means and the time information corresponding thereto, and the gesture output from the gesture analyzing means. Based on the type and time information corresponding to it and the information of the pointing object, search the word or phrase corresponding to the gesture from the semantic structure of the voice recognition result, and the word or phrase corresponding to the retrieved gesture Detecting a temporal relationship with the information of the pointing object and generating a semantic structure in which the semantic structure of the voice recognition result and the semantic structure of the gesture type are integrated based on the detected temporal relationship. Output. Therefore, it is possible to provide a multi-modal information integrated analysis device capable of comprehensively analyzing a voice uttered by a human and a gesture of the human and outputting an analysis result. This makes it possible to analyze and judge more complicated and concrete input information of the human based on the voice spoken by the human and the gesture of the human.

[Brief description of the drawings]

FIG. 1 is a block diagram of a multi-modal information integrated analysis device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a monitoring control process executed by a monitoring controller of FIG.

FIG. 3 is a flowchart showing a gesture analysis process executed by the gesture analysis unit of FIG.

FIG. 4 is a timing chart showing collection of voice and gesture information executed in the multimodal information integrated analysis device of FIG. 1.

5 is a front view showing a process of gesture recognition and selection of a pointing object, which is executed in the multimodal information integrated analysis device of FIG. 1. FIG.

FIG. 6 is a front view showing an example of a screen of the CRT display of FIG.

FIG. 7 is a flowchart showing a gesture analysis process of a modified example executed by the gesture analysis unit of FIG.

FIG. 8 is a diagram showing a minimum rectangle in one process of the gesture analysis process of FIG.

9 is a diagram showing a plurality of angles between two gesture lines in one process of the gesture analysis process of FIG. 7. FIG.

FIG. 10 is a diagram showing a process of marking a gesture in the gesture analysis process of FIG.

FIG. 11 is a diagram showing a length ratio in one process of the gesture analysis process of FIG. 7.

FIG. 12 is a diagram showing the position of a peak point of a gesture in one process of the gesture analysis process of FIG. 7.

FIG. 13 is a diagram showing an upper 25% region in one process of the gesture analysis process of FIG. 7.

14 (a), (b), (c) and (d) are shown in FIG.
FIG. 6 is a diagram showing a process of determining “encircling”, “drawing a line”, and “scrambling” in the gesture analysis process of FIG.

FIG. 15 “Draw a line” of the gesture analysis process of FIG.
FIG. 6 is a diagram showing a process of a process of determining “circle” and “encircle”.

FIG. 16 is a block diagram of a conventional multi-modal interactive geographic guidance system.

[Explanation of symbols]

10 ... Monitoring control unit, 11 ... Voice recognition unit, 12 ... Language analysis unit, 13 ... Graphic user interface control unit (G
UI control unit), 14 ... Gesture analysis unit, 15 ... Integrated analysis unit, 21 ... HM network, 22 ... Context-free grammar, 23 ... Word dictionary, 24 ... Map database, 25 ... Gesture dictionary, 30 ... Clock signal generator, 31 ... Microphone, 32 ... Keyboard, 32a ... Start button, 32b ... Stop button, 32c ... Quit button, 33, 35 ... CRT display, 34 ... Mouse.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Internal reference number FI Technical display location G10L 3/00 571 G06F 15/38 Z 15/62 322M (72) Inventor Kyunho Loken Kim Kyoto Shiraka-gun, Seika-cho, Ina-kan, Osamu, 5 Sanhira-tani Co., Ltd. ATR Co., Ltd. Voice Translation Research Laboratory (72) Inventor, Mutsuko Tomoyo Kyoto, Soraku-gun, Seika-cho, Inaiya, 5 Sanpei, A (72) Inventor, Takuma Morimoto, Soka Town, Kyoto Prefecture, Osamu Osamu, Osamu Osamu, No. 5, Mihiraya, Tokyo, Inc.

Claims (4)

[Claims] 1. A time measuring means for outputting time information elapsed from a predetermined reference time, and voice recognition of an inputted uttered voice based on the time information outputted from the time measuring means, and a voice recognition result is obtained. , A speech recognition unit that outputs together with time information corresponding to the speech recognition result, and a language analysis using knowledge about a predetermined language based on the speech recognition result output from the speech recognition unit and the corresponding time information. Then, the semantic structure of the voice recognition result is displayed on the screen with a language analysis means for outputting together with the corresponding time information and a plurality of pointing object candidates, and a human gesture is displayed on the displayed screen. Based on the input means for inputting and the time information output from the timing means, the position on the screen of the trajectory of the gesture input via the input means, and By analyzing the interface control means that outputs together with the corresponding time information and the position on the screen of the trajectory of the gesture output from the interface control means by using the knowledge about the diagram including the plurality of pointer candidates, ,
Type of the above gesture and time information corresponding to it,
Gesture analysis means for outputting information of a pointing object which is a pointing object candidate pointed by the gesture among the plurality of pointing object candidates, and a semantic structure of the speech recognition result output from the language analyzing means and its correspondence Based on the time information, the type of the gesture output from the gesture analysis means, the time information corresponding to it, and the information of the pointing object, the word or phrase corresponding to the gesture from the semantic structure of the voice recognition result. And detecting the temporal relationship between the searched word or phrase corresponding to the gesture and the information of the pointing object, and based on the detected temporal relationship, the semantic structure of the voice recognition result and the A multi-modal information integrated solution characterized by having an integrated analysis means for generating and outputting a semantic structure in which the semantic structures of the types of gestures are integrated Analyzer. 2. The multimodal information integrated analysis device according to claim 1, wherein the word corresponding to the gesture is a demonstrative word. 3. The gesture types analyzed by the gesture analyzing means include a “circle” gesture, a “draw line” gesture, a “dotting” gesture, a “marking” gesture, and a random gesture. 3. The multimodal information integration analysis device according to claim 1, further comprising a "scrambling" gesture which is a depiction with various movements. 4. The gesture analysis means divides the rectangle into a plurality of areas by a plurality of lines passing through a center of a rectangle surrounding the trajectory of the gesture, and establishes a relationship between the divided areas and the trajectory of the gesture. The type of the gesture is determined based on the gesture.
Alternatively, the multimodal information integrated analysis device described in 3.