JP7414231B2 - Multimodal speech recognition device and multimodal speech recognition method - Google Patents

Description

The present invention relates to multimodal speech recognition technology that recognizes speech using speech and a gaze point image at the time of speech production.

With the development of sensing technology, it has become possible to acquire various signals simultaneously. Against this background, in the technical field of speech recognition, multimodal speech recognition technology has been proposed that recognizes speech using speech and information other than speech in order to improve speech recognition performance.
For example, Non-Patent Document 1 discloses a voice recognition technology that uses voice and a lip image that shows the movement of the mouth when making a voice.
Further, End-to-End speech recognition technology based on deep learning (deep learning of neural networks) is disclosed in Non-Patent Document 2.

“Multimodal speech recognition using depth images of lips”, Shohei Oshio et al., Information Processing Society of Japan Research Report, Vol. 2014-SLP-102-No. 2, 2014/7/24 “Transition and cutting edge of speech recognition technology”, Tatsuya Kawahara, Journal of the Acoustical Society of Japan, Vol. 74, No. 7 (2018), pp. 381-386

Non-Patent Document 1 discloses that speech recognition performance is improved by using lip images together, but does not disclose the use of information other than lips.
The end-to-end speech recognition technology based on deep learning disclosed in Non-Patent Document 2 is a series conversion model that directly maps a feature sequence (feature vector) obtained from speech to a character string. (Encoder-Decoder). In recent years, attempts have been made to improve speech recognition performance by combining an attention mechanism with a sequence transformation model and assigning weights to feature vectors. However, Non-Patent Document 2 does not disclose combining an attention mechanism with multimodal speech recognition.
As a result of various studies on techniques for improving speech recognition performance, the inventor found that when working while uttering a voice, the voice and the point of interest are mutually related. It has been found that speech recognition performance can be improved by estimating correlations.
The present invention was devised in view of these points, and is a multimodal speech recognition technology that improves speech recognition performance by recognizing speech using speech and images of the gazing point around the gazing point. The purpose is to provide.

The first invention relates to a multimodal speech recognition device.
The first aspect of the present invention includes a voice information input means, a voice characteristic information extraction means, a gaze point image information input means, a gaze point characteristic information extraction means, a storage means, and a conversion means.
The voice information input means inputs voice information indicating the voice of the speaker. As the voice information input means, various voice information input means capable of inputting voice information can be used. Preferably, audio information input means including a microphone that converts audio into electrical signals is used. Note that it is also possible to use an audio information input means that includes a storage medium in which audio information is stored in advance.
The audio feature information extraction means extracts audio feature information in time series from the audio information input from the audio information input means. As the audio feature information extraction means, a convolutional neural network (CNN) having a convolution layer and a pooling layer is preferably used.
The gazing point image information input means inputs gazing point image information indicating a gazing point image around the gazing point that the speaker is gazing at when uttering voice. As the gaze point image information input means, various gaze point image information input means that can input gaze point images can be used. Preferably, a gaze point image information input means including a line of sight measuring device is used. Preferably, a line-of-sight measuring device is used that can output gaze point position information indicating the position of the gaze point in a subjective image captured by a camera built into the device.
The point of interest feature information extraction means extracts point of interest feature information in time series from the point of interest image information input from the point of interest image information input means. Preferably, the gaze point image information is extracted in synchronization with the audio feature information. As the gaze point feature information extraction means, a multilayer neural network (CNN) is used, similar to the audio feature information extraction means.
The storage means stores character string information. The character string information includes hiragana, katakana, numbers, commonly used kanji, etc. that form text information such as documents.
The conversion means converts text information corresponding to the time-series voice feature information extracted by the voice feature information extraction means and the time-series gaze point feature information extracted by the gaze point feature information extraction means into characters stored in the storage means. It is formed using character string information selected from string information. Preferably, the text information is output from an output means such as a display means.
Any appropriate method can be used to form text information corresponding to time-series voice feature information and time-series gaze point feature information.
The audio feature information extraction means, the gaze point feature information extraction means, and the conversion means can be configured by one computer or by separate computers. Moreover, it can also be configured to be located far away and connectable via a communication line such as the Internet.
In the first invention, since speech is recognized using time-series voice feature information and time-series gaze point image information, it is possible to improve speech recognition performance.
In a different form of the first invention, the converting means includes a voice feature information encoding means, a voice code weighting means, a gaze point characteristic information encoding means, a gaze point code weighting means, and a decoding means.
The audio feature information encoding means encodes the time-series audio feature information extracted by the audio feature information extraction means and outputs a time-series audio code. A recurrent neural network (RNN) is preferably used as the audio feature information encoding means. For example, Bi-directional Long Short Term Memory (BLSTM), which is a form of recurrent neural network (RNN), is used.
The speech code weighting means weights the time-series speech codes output from the speech feature information encoding means and outputs time-series weighted speech codes. As the speech code weighting means, it is possible to use speech code weighting means having various configurations that can appropriately apply weights to time-series speech codes.
The gaze point feature information encoding means encodes the time series gaze point feature information extracted by the gaze point feature information extraction means and outputs a time series gaze point code. As the gaze point feature information encoding means, bidirectional long-term short-term memory (BLSTM), which is a form of recurrent neural network (RNN), or the like is used, similar to the audio feature information encoding means.
The point-of-regard code weighting means weights the time-series point-of-regard code output from the point-of-regard characteristic information encoding means, and outputs a time-series weighted point-of-regard code. As the point of interest code weighting means, it is possible to use point of interest code weighting means having various configurations that can appropriately apply weights to time-series point of interest codes.
The decoding means converts the time-series weighted speech code output from the speech code weighting means and the time-series weighted attention point code output from the attention point code weighting means into an integrated code (weighted integrated code). Corresponding text information is formed by character string information selected from character string information stored in the storage means.
A recurrent neural network (RNN) is preferably used as the decoding means. For example, long short-term memory (LSTM), which is a form of recurrent neural network (RNN), is used.
The audio feature information encoding means, the audio code weighting means, the attention point feature information encoding means, the attention point code weighting means, and the decoding means can also be configured by one computer. It can also be configured by separate computers. Moreover, it can also be configured to be located far away and connectable via a communication line such as the Internet.
In this embodiment, it is possible to accurately estimate the correlation between audio information and gaze point image information.
In a different form of the first invention, an attention mechanism (Attention) of a sequence conversion model constituted by a neural network is used as the voice code weighting means and the point of interest code weighting means.
In this embodiment, appropriate weights can be given to time-series speech codes and time-series gaze point codes, and speech recognition performance can be reliably improved.
In a different form of the first invention, the point of interest image information input means can input point of interest position information indicating the position of the point of interest in the subjective image of the speaker (the imaging area of the point of interest image information input means). The text information is then displayed on the display means in association with the position of the speaker's point of interest indicated by the point of interest position information. For example, text information is displayed near the position of the point of interest in the subjective image captured by the point of interest image information input means displayed on the display means.
In this embodiment, the text information is displayed on the display means in association with the position of the point of interest, so the voice uttered by the speaker and the point of interest of the speaker can be easily confirmed.
In a different form of the first invention, when text information displayed on the display means is selected, audio information corresponding to the selected text information (audio information input when recognizing the text information) is output. It is configured as follows.
In this embodiment, the voice uttered by the speaker can be easily confirmed.
The second invention relates to a multimodal speech recognition method.
The present invention has first to third steps.
In the first step, voice feature information is extracted in time series from voice information indicating the voice of the speaker. The process of the first step is executed, for example, by the audio feature information extraction means of the first invention.
In the second step, gaze point feature information is extracted in time series from gaze point image information indicating gaze point images around the gaze point that the speaker is gazing at. The process of the second step is executed, for example, by the gaze point feature information extraction means of the first invention.
In the third step, text information corresponding to the extracted time-series voice feature information and the extracted time-series gaze point feature information is formed using string information selected from string information stored in the storage means. do. The processing of the third step is executed, for example, by the conversion means of the first invention. Preferably, the text information is output from an output means such as a display means.
The second invention has the same effects as the first invention.
In a different form of the second invention, the third step includes fourth to eighth steps.
In the fourth step, the extracted time-series audio feature information is encoded to output a time-series audio code. The process of the fourth step is executed, for example, by the audio feature information encoding means of the first invention.
In the fifth step, weights are given to the time-series speech codes and time-series weighted speech codes are output. The processing of the fifth step is executed, for example, by the speech code weighting means of the first invention.
In the sixth step, the time-series gaze point feature information is encoded and a time-series gaze point code is output. The process of the sixth step is executed, for example, by the gaze point feature information encoding means of the first invention.
In the seventh step, weights are given to the time-series gazing point codes to output time-series weighted gazing point codes. The process of the seventh step is executed, for example, by the point-of-regard code weighting means of the first invention.
In the eighth step, the text information corresponding to the integrated code (weighted integrated code) that integrates the time-series weighted speech code and the time-series weighted gaze point code is added to the text information stored in the storage means. Formed by character string information selected from among them. The process of the eighth step is executed, for example, by the decoding means of the first invention.
Preferably, the voice code weighting process in the fifth step and the gaze point code weighting process in the seventh step are performed by an attention mechanism of a sequence conversion model configured with a neural network.
This embodiment has the same effects as the first invention.

The multimodal speech recognition device and multimodal speech recognition method of the present invention can improve speech recognition performance by recognizing speech using speech and a gaze point image around the gaze point.

FIG. 1 is a block diagram of an embodiment of a multimodal speech recognition device of the present invention. FIG. 3 is a diagram illustrating the operation of a voice feature extracting means of a multimodal voice recognition device according to an embodiment. FIG. 3 is a diagram illustrating the operation of a voice feature encoding means of a multimodal voice recognition device according to an embodiment. FIG. 3 is a diagram illustrating the operation of a speech code weighting means of a multimodal speech recognition device according to an embodiment. FIG. 3 is a diagram illustrating the operation of a gaze point image feature extracting means of the multimodal speech recognition device according to one embodiment. FIG. 3 is a diagram illustrating the operation of a gaze point image feature amount encoding means of the multimodal speech recognition device of one embodiment. FIG. 3 is a diagram illustrating an operation of a gaze point code weighting means of a multimodal speech recognition device according to an embodiment. FIG. 3 is a diagram illustrating the operation of a decoding means of a multimodal speech recognition device according to an embodiment. FIG. 1 is a diagram illustrating the configuration of a multimodal speech recognition device according to an embodiment. FIG. 3 is a diagram illustrating the operation of a speech code weighting means of a multimodal speech recognition device according to an embodiment. FIG. 6 is a diagram illustrating the operation of the point-of-regard code weighting means of the multimodal speech recognition device according to one embodiment. It is a figure which shows the example of a display of a display means.

Embodiments of the present invention will be described below with reference to the drawings.
A block diagram of one embodiment of the multimodal speech recognition device of the present invention is shown in FIG.
The multimodal speech recognition device of this embodiment uses an end-to-end speech recognition framework based on deep learning. Then, using a sequence transformation model that has multiple attention mechanisms, the voice information indicating the voice of the speaker and the image of the gaze point around the gaze point of the speaker who is uttering the voice are integrated. Based on the correlation, audio information is converted into text information formed by character string information.

The multimodal speech recognition device of this embodiment includes a processing means 10, a speech information input means 30, a gaze point image information input means 40, a storage means 50, a display means 60, and the like.

The voice information input means 30 inputs voice information indicating the voice uttered by the speaker. The audio information may be an audio waveform or a spectrum (frequency information). Preferably, the audio information input means 30 includes a microphone and an AD conversion means. Of course, as the voice information input means 30, various voice information input means capable of inputting voice information can be used.
The gazing point image information input means inputs gazing point image information indicating a gazing point image around a gazing point that the speaker is gazing at while uttering a voice. As the gaze point image information input means, for example, a line of sight measuring device that can be worn by the speaker can be used. As the gaze point image around the gaze point, an image around the gaze point (for example, a rectangular area of a predetermined pixel size centered on the gaze point) in the subjective image captured by the line of sight measurement device can be used. In this case, preferably, an image information extraction means is provided that extracts image information around the gaze point from the subjective image captured by the line of sight measuring device.
Preferably, a line-of-sight measuring device is used that can output gaze point position information indicating the position of the speaker's gaze point in a subjective image captured by a gaze measuring device worn on the speaker.
The position of the gaze point in the subjective image captured by the line of sight measuring device corresponds to the "position of the speaker's gaze point" of the present invention.
The storage means 50 stores character string information. The character string information includes hiragana, katakana, numbers, commonly used kanji, etc. that form text information such as documents.
The display means 60 is used to display speech recognition results and the like.

The processing means 10 includes a conversion means 20, an audio feature amount extraction means 110, and a gaze point image feature amount extraction means 140.
As shown in FIG. 2, the audio feature extraction means 110 extracts audio features (speech feature vectors) X1 to Xn in time series from the audio information input from the audio information input means 30. Note that the audio section of the audio information (the section presumed to be a semantic word or sentence unit) can be determined by the presence of silent sections before and after it. The audio feature extraction means 110 outputs an audio feature sequence (an audio feature vector sequence) {X1, X2, . . . , Xn}.
In this embodiment, a convolutional neural network (CNN) having a convolution layer and a pooling layer is used as the audio feature amount extraction means 110.
The audio feature extraction means 110 corresponds to the "audio feature information extraction means" of the present invention, and the audio feature series (audio feature vector series) {X1, X2, ..., Xn} corresponds to the "audio feature information extraction means" of the present invention. It corresponds to "time-series audio feature information".

As illustrated in FIG. Extract the quantity vector) Y1 to Ym. The extraction process of time-series gazing-point image information by the gazing-point image feature extracting means 140 is performed in synchronization with the extraction process of time-series audio feature information by the audio feature extracting means 110. Note that although there may be a case where the voice utterance and the gaze movement by the speaker are out of sync, the correspondence relationship is estimated by the processing of the conversion means 20, so they do not need to be completely synchronized. The gazing point image feature amount extracting means 140 outputs a gazing point image feature amount series (a gazing point image feature vector sequence) {Y1, Y2, . . . , Ym}.
In this embodiment, a convolutional neural network (CNN) is used as the gaze point image feature extraction means 140, similar to the audio feature extraction means 110.
The gazing point image feature extracting means 140 corresponds to the ``gazing point feature information extraction means'' of the present invention, and includes the gazing point image feature amount series (the gazing point image feature vector series) {Y1, Y2, . . . , Ym } corresponds to "time-series gaze point feature information" of the present invention.

The conversion means 20 converts the audio feature series {X1, Text information, which is a voice recognition result, is formed using a character string selected from among the character strings stored in the storage means 50 based on the gazing point image feature amount).
In this embodiment, the converting means 20 includes an audio feature encoding means 120, an audio code weighting means 130, a gaze point image feature encoding means 150, a gaze point code weighting means 160, an integrating means 170, and a decoding means 180. have.

As shown in FIG. 3, the audio feature encoding means 120 converts the time-series audio features X1 to Xn (speech feature series {X1, X2, . . . , Xn}) and outputs time-series speech codes (speech code vectors) h1 to hn. That is, the audio feature encoding means 120 outputs an audio code sequence (an audio code vector sequence) {h1, h2, . . . , hn}.
In this embodiment, as shown in FIG. 9, bidirectional long-term short-term memory (BLSTM), which is a form of recurrent neural network (RNN), is used as the audio feature encoding means 120.
The audio feature encoding means 120 corresponds to the "audio feature information encoding means" of the present invention, and the audio code series (voice code vector series) {h1, h2, ..., hn} corresponds to the "audio feature information encoding means" of the present invention. It corresponds to "time series audio code".

The audio code weighting means 130 gives weights to the time-series audio codes h1 to hn (speech code series {h1, h2, ..., hn}) output from the audio feature encoding means 120, and converts them into time-series audio codes. The weighted audio codes (weighted audio code vectors) (a1*h1) to (an*hn) are output. That is, the audio code weighting means 130 outputs a weighted audio code sequence (weighted audio code vector sequence) {a1*h1, a2*h2, . . . , an*hn}. Note that the weights a1 to an are set so that the sum of the weights a1 to an is "1".
In this embodiment, bidirectional long-term short-term memory (BLSTM), which is a form of recurrent neural network (RNN), is used as the speech code weighting means 130.
The operation of the voice code weighting means 130 will be described later.

As shown in FIG. The sequence {Y1, Y2, . . . , Ym}) is encoded to output time-series gazing point codes (gazing point code vectors) s1 to sm. That is, the gazing point image feature amount encoding means 150 outputs the gazing point code series (the gazing point code vector series) {s1, s2, . . . , sm}.
In this embodiment, bidirectional long-term short-term memory (BLSTM), which is a form of recurrent neural network (RNN), is used as the gaze point image feature encoding means 150.
The gazing point image feature amount encoding means 150 corresponds to the ``gazing point feature information encoding means'' of the present invention, and the gazing point code series (the gazing point code vector series) {s1, s2, . . . , sm} is , corresponds to the "time-series gaze point code" of the present invention.

The point-of-regard code weighting means 160 weights the time-series point-of-regard codes s1 to sm output from the point-of-regard image feature encoding means to create a time-series weighted point-of-regard code (weighted point-of-regard vector). )(b1*s1) to (bm*sm) are output. That is, the point-of-regard code weighting means 160 outputs a weighted point-of-regard code sequence (weighted point-of-regard code vector sequence) {b1*s1, b2*s2, . . . , bm*sm}. Note that the weights b1 to bm are set so that the sum of the weights b1 to bm is "1".
In the present embodiment, an attention mechanism (Attention) of a sequence conversion model constituted by a neural network is used as the gaze point code weighting means 160.
The operation of the gaze point code weighting means 160 will be described later.

The integrating means 170 outputs time-series weighted speech codes (a1*h1) to (an*hn) (weighted speech code sequences {a1*h1, a2*h2, . . . ) output from the speech code weighting means 130. , an*hn}) and the weighted point of interest code (b1*s1) to (bm*sm) (weighted point of interest code sequence {b1*s1, b2*s2, . ..., bm*sm}) to create a time series integrated weighted code (a1*h1+b1*s1) ~ (an*hn+bm+sm) (weighted code sequence {a1*h1+b1*s1, a2*h2+b2*s2, ..., an*hn+bm*sm}) is output.

The decoding means, as shown in FIG. h1+b1*s1, a2*h2+b2*s2, . . . Formed by
As a method for selecting the character string information C1 to Ci, for example, the code output from the hidden layer of each LSTM at each time is converted into an output score (probability value) of the character string using a Softmax function. Then, a method is used in which a character string with a high output score is selected.
The text information (speech recognition result) decoded by the decoding means 180 is displayed on the display means 60.
In this embodiment, a long short term memory (LSTM), which is a form of a recurrent neural network (RNN), is used as the decoding means 180.

This embodiment has an audio channel that processes audio information and a gaze point image channel that processes gaze point images. The audio channel includes audio information input means 30, audio feature extraction means 110, audio feature encoding means 120, and audio code weighting means 130. The gazing point image channel includes a gazing point image information input means 40 , a gazing point image feature extracting means 140 , a gazing point image feature encoding means 150 , and a gazing point code weighting means 160 .
Audio feature extracting means 110, audio feature encoding means 120, audio code weighting means 130, gazing point image feature extracting means 140, gazing image feature encoding means 150, gazing point code weighting means 160, integrating means 170 The decoding means 180 can be configured in a common computer or can be configured in separate computers.
Further, it is also possible to arrange one means far away from the other means and to transmit and receive information between the two means via a communication line such as the Internet.

Next, the learning operation of the multimodal speech recognition device of this embodiment will be explained.
In learning, the transformation means 20 ( Iteratively learns various weight parameters of the neural network that makes up the series conversion model. For example, audio information such as "Select circuit breaker 745" and "point-of-regard image information series when selecting circuit breaker 745" are input. Then, various weight parameters of the converting means 20 are learned so that the error between the text information output from the decoding means 180 and the input voice information "Select circuit breaker 745" is minimized.

Next, the speech recognition operation of the multimodal speech recognition device of this embodiment will be explained.
When the audio information and the gaze point image information are input, the processing of the processing means 10 of the present invention is started. The processing operation of the processing means 10 is as described above.
Here, the weighting operation by the voice code weighting means 130 will be explained with reference to FIG. 10.
End-to-end speech recognition technology based on deep learning includes a decoder that decodes a speech feature sequence (speech feature vector) obtained from speech into a character string. The encoder converts the audio feature sequence into a hidden state vector, and the decoder converts the code sequence into text information that is a recognition result via the hidden state vector.
In this embodiment, as shown in FIG. 9, an audio encoder that encodes an audio feature sequence, a gaze encoder that encodes a gaze image feature sequence (a gaze image feature vector), an audio code It has a decoder that integrates the sequence (voice code vector) and the point-of-interest code sequence (point-of-interest code vector) and maps it to a character string. Further, in this embodiment, the decoder generates an integrated weighted code sequence by assigning weights (attention) to each of the voice code sequence and the point-of-regard code sequence. Note that the audio encoder includes audio feature encoding means 120. Furthermore, the gaze point encoder is configured by gaze point image feature amount encoding means 150. Further, the decoder includes a voice code weighting means 130, a point of interest code weighting means 160, an integrating means 170, and a decoding means 180.
The speech code weight (speech code weight vector) in the time section at an arbitrary time t of the decoding means 180, which is shown by a dashed line in FIG. {h(1),...,h(n)}, which is a hidden state vector sequence of dynamically assigned based on For example, the degree of similarity a(i) (i=1,...,n) between the hidden state vector u(t-1) and each element of the hidden state vector sequence {h(1),...,h(n)} is , by taking the inner product of the hidden state vector u(t-1) and each element of the hidden state vector sequence {h(1),...,h(n)}, a(i)=u(t-1)・It can be determined numerically as h(i) (i=1,...,n). Here, the audio code weights a(i) are normalized so that the total becomes "1". The input from the speech encoder to the decoder at time (t) uses the speech code weight a(i) (i=1,...,n) and the hidden state vector sequence {h(1),...,h(n)}. is expressed as [a(1)*h(1)+...+a(n)*h(n)].
Note that various evaluation scales can be used in calculating the degree of similarity.

The weighting operation by the gaze point code weighting means 160 will be explained with reference to FIG. 11.
The point of interest code weight (point of interest code weight vector) in the time section at an arbitrary time t of the decoding means 180, which is shown by a dashed line in FIG. The hidden state vector sequence {s(1),...,s(m)} output from the layer and the hidden state vector u( t-1). For example, the similarity b(j) (j=1,...,m) between the hidden state vector u(t-1) and each element of the hidden state vector sequence {s(1),...,s(m)} is , by taking the inner product of the hidden state vector u(t-1) and each element of the hidden state vector sequence {s(1),...,s(m)}, b(j)=u(t-1)・It can be determined numerically as s(j) (j=1,...,m). Here, the gaze point code weights b(j) are normalized so that the total becomes "1". The input from the gaze point encoder to the decoder at time (t) is the gaze point code weight b(j) (j=1,...,m) and the hidden state vector sequence {s(1),...,s(m)} is expressed as [b(1)*s(1)+...+b(m)*s(m)].
Note that various evaluation scales can be used in calculating the degree of similarity.

In the manner described above, weights can be dynamically assigned to the output codes from the speech encoder and the gaze point encoder.
Next, in the integrating means 170, the weighted speech code sequence and the gaze point code sequence obtained by the above method are converted into r(t)=[a(1)*h(1)+...+a(n)*h( n)]+[b(1)*s(1)+...+b(m)*s(m)], and converts this r(t) into the LSTM of the decoding means 180 (decoder) at time t. Use as input.
Then, as described above, the decoding means 180 converts the character string output from each LSTM at each time into an output score (probability value) of the character string using the Softmax function. Then, character strings with high output scores are selected to form text information (speech recognition results).

As described above, by weighting the audio code sequence by the audio code weighting means 130 (code weighting in the audio channel) and weighting the gazing point code sequence by the gazing point code weighting means 160 (weighting in the gazing point image channel), Text information (character string information forming the text information) corresponding to the integrated weighted code sequence input to the decoding means 180 can be formed while estimating the correlation between the audio information and the gaze point image information.
The voice uttered by the speaker and the speaker's gaze point when uttering the voice are mutually related.
Therefore, in this embodiment, the speech recognition performance can be improved by performing speech recognition by estimating the correlation between the speaker's voice and the point of interest.

In order to confirm the effectiveness of this embodiment, we conducted a comparative experiment using only audio information (Model 1) and using audio information and gaze point image information (Model 2). CER (Character Error Rate) was calculated. Note that CER is expressed as [CER=(S+D+I)*100/N]. Here, S represents the number of substitution errors, D represents the number of omission errors, I represents the number of insertion errors, and N represents the number of characters in the correct sentence.
As a result of the experiment, the CER was 7.2% in (Model 1), but it was reduced to 6.9% in (Model 2), confirming the effect of applying the configuration of the present invention.

In the embodiments described above, an integrated code sequence (integrated code vector) r ([r=a*h+b *s]), but it is also possible to change the fusion ratio of the speech code sequence h and the gaze point code sequence s. For example, an integrated code sequence r expressed as [r=a*h+g*(b*s)] can be used. Here, g is a fusion weight (fusion weight vector) indicating the fusion rate of the gaze point code series. The fusion weight vector may be fixed or dynamically assigned.

Next, a method for outputting speech recognition results will be explained.
In this embodiment, the text information decoded by the decoding means 180 is displayed on the display means 60.
FIG. 12 shows an example of a display screen 200 that displays text information.
A display screen 200 shown in FIG. 12 displays an operation panel 300 for turning on and off circuit breakers and line switches. The operation panel 300 includes circuit breaker selection buttons 311 to 313 that are operated when selecting circuit breakers 740, 742, and 745, and line switch selection buttons 314 to 314 that are operated when selecting line switches 740, 742, and 745. 316, an on button 317 that is operated to turn on the power, and an off button 318 that is operated to shut off the power are provided.
Here, the speaker utters the voice "Line switch 745 selection operation" and looks at the line switch selection button 316 on the operation panel 300, and the text information "Line switch 745 selection operation" is output from the decoding means 180. It is assumed that voice recognition has been performed. In this embodiment, on the display screen 200 on which the operation panel 300 is displayed, text information such as "Line switch button 745 operation selection" is displayed at a position related to the gaze point, which in FIG. 12 corresponds to the line switch selection button 316. is displayed.
Thereby, the speaker's voice and the position of the gaze point can be easily distinguished.
In FIG. 12, after uttering the voice ``Select operation of line switch 745'', the voice ``Turn it on'' is uttered, and as the user gazes at the ON button 317 on the operation panel 300, the line switch 745 is uttered. In FIG. 12, text information "Enter" is displayed at the location corresponding to the enter button 317.
Note that in a state where text information is displayed on the display screen 200, by selecting (for example, touching) the text information displayed on the display screen 200, audio information corresponding to the displayed text information, For example, it may be configured such that audio information input when recognizing the text information is output from audio output means such as a speaker.
Processing for displaying text information on the display means 60, processing for outputting audio information corresponding to the text information from the audio output means, etc. are executed by the processing means 10, for example.

Although the multimodal speech recognition device has been described above, the present invention can also be configured as a multimodal speech recognition method.
(Aspect 1)
A multimodal speech recognition method, comprising:
A first step of extracting voice feature information in time series from voice information indicating the voice of the speaker;
a second step of extracting gaze point feature information in time series from gaze point image information indicating gaze point images around the gaze point that the speaker is gazing at;
A third step of forming text information corresponding to the extracted time-series voice feature information and the extracted time-series gaze point feature information using character string information selected from character string information stored in the storage means. A multimodal speech recognition method comprising steps.
(Aspect 2)
A multimodal speech recognition method according to aspect 1, comprising:
The third step is
a fourth step of encoding the extracted time-series audio feature information and outputting a time-series audio code;
a fifth step of adding weight to the time-series audio code and outputting a time-series weighted audio code;
a sixth step of encoding the time-series gaze point feature information and outputting a time-series gaze point code;
a seventh step of adding a weight to the time-series point-of-regard code and outputting a time-series weighted point-of-regard code;
Text information corresponding to an integrated code that integrates the time-series weighted speech code and the time-series weighted gaze point code is formed using character string information selected from character string information stored in a storage means. A multimodal speech recognition method, comprising: an eighth step of:
Such a multimodal speech recognition method also has the same effects as the multimodal speech recognition device described above.

The present invention is not limited to the configuration described in the embodiments, and various changes, additions, and deletions are possible.
In the embodiment, audio information and gaze point image information are used as multimodal information, but three or more pieces of information can also be used as multimodal information. For example, audio information, gaze point image information, and gesture information (gestures and hand gestures) can also be used as multimodal information.
Although the gaze point image information is input using the visible light region image sensor of the line of sight measuring device, the gaze point image information can be input using various sensors such as an infrared sensor or an ultraviolet sensor.
The multimodal speech recognition device and multimodal speech recognition method of the present invention are not limited to confirmation of operations by workers, but can be used in various fields such as creating subtitles for videos with audio, video search, skill inheritance using videos, and education and training. It can be used in
As the voice information input means, voice information input means having various configurations capable of inputting voice information can be used. Furthermore, a storage means or the like that stores voice information in advance can also be used as the voice information input means.
As the gaze point image information input means, gaze point image information input means having various configurations capable of inputting gaze point image information around the gaze point can be used. Furthermore, a storage device or the like that stores point-of-regard image information in advance can also be used as the point-of-focus image information input device.
Audio feature extraction means (audio feature information extraction means), audio feature encoding means (audio feature information encoding means), audio code weighting means, gaze point image feature extraction means (gazing point feature information extraction means), notes The configurations of the viewpoint image feature amount encoding means (point of interest characteristic information encoding means), the point of attention code weighting means, the integrating means, and the decoding means are not limited to the configurations described in the embodiments.
The method of displaying the voice recognition results etc. on the display means is not limited to the method described in the embodiment.
The method of outputting the voice recognition results etc. is not limited to the method of displaying them on the display means. For example, it is also possible to use a method of transmitting the information to a remote management device via a communication line.

10 Processing means 20 Conversion means 30 Voice information input means 40 Gaze point image information input means 50 Storage means 60 Display means 110 Voice feature amount extraction means (voice feature information extraction means)
120 Voice feature encoding means (voice characteristic information encoding means)
130 Audio code weighting means 140 Gazing point image feature extraction means (gazing point feature information extraction means)
150 Gaze point image feature amount encoding means (gazing point feature information encoding means)
160 Gaze point code weighting means 170 Integrating means 180 Decoding means 200 Display screen 300 Operation panel 311 to 313 Circuit breaker selection buttons 314 to 316 Line switch selection button 317 On button 318 Off buttons 321, 322 Text information display section

Claims (7)

A multimodal speech recognition device,
voice information input means for inputting voice information indicating the voice of the speaker;
Gaze point image information input means for inputting gaze point image information indicating a gaze point image around the gaze point that the speaker is gazing at;
a storage means for storing character string information;
audio feature information extraction means for extracting audio feature information in chronological order from the audio information input from the audio information input means;
Gaze point feature information extraction means for extracting gaze point feature information in chronological order from the gaze point image information input from the gaze point image information input means;
Text information corresponding to the time series voice feature information extracted by the voice feature information extraction means and the time series gaze point feature information extracted by the gaze point feature information extraction means is stored in the storage means. 1. A multimodal speech recognition device comprising: conversion means for forming character string information selected from character string information. The multimodal speech recognition device according to claim 1,
The conversion means is
audio feature information encoding means for encoding each of the time-series audio feature information extracted by the audio feature information extraction means and outputting a time-series audio code;
audio code weighting means for weighting each of the time-series audio codes output from the audio feature information encoding means to output a time-series weighted audio code;
gaze point feature information encoding means for encoding each of the time series gaze point feature information extracted by the gaze point feature information extraction means and outputting a time series gaze point code;
a point of interest code weighting means for weighting each of the time-series point-of-regard codes outputted from the point-of-gaze characteristic information encoding means to output a time-series weighted point-of-regard code;
Text information corresponding to an integrated code obtained by integrating the time-series weighted audio code output from the audio code weighting means and the time-series weighted attention point code output from the attention point code weighting means is stored in the memory. 1. A multimodal speech recognition device comprising decoding means for decoding information selected from character string information stored in the means. The multimodal speech recognition device according to claim 2,
A multimodal speech recognition device characterized in that an attention mechanism of a sequence conversion model constituted by a neural network is used as the speech code weighting means and the point-of-regard code weighting means. A multimodal speech recognition device according to any one of claims 1 to 3, comprising:
Equipped with display means,
The point-of-gaze image information input means is capable of inputting point-of-gaze position information indicating the position of the point of interest in the subjective image of the speaker;
A multimodal speech recognition device characterized in that the text information is displayed on the display means in association with the position of the speaker's point of interest indicated by the point of interest position information. The multimodal speech recognition device according to claim 4,
A multimodal speech recognition device characterized in that, when the text information displayed on the display means is selected, the multimodal speech recognition device is configured to output speech information corresponding to the selected text information. A multimodal speech recognition method, comprising:
A first step of extracting voice feature information in time series from voice information indicating the voice of the speaker;
a second step of extracting gaze point feature information in time series from gaze point image information indicating gaze point images around the gaze point that the speaker is gazing at;
A third step of forming text information corresponding to the extracted time-series voice feature information and the extracted time-series gaze point feature information using character string information selected from character string information stored in the storage means. A multimodal speech recognition method comprising steps. The multimodal speech recognition method according to claim 6,
The third step is
a fourth step of encoding each of the extracted time-series audio feature information and outputting a time-series audio code;
a fifth step of assigning a weight to each of the time-series audio codes and outputting a time-series weighted audio code;
a sixth step of encoding each of the time-series gaze point feature information and outputting a time-series gaze point code;
a seventh step of assigning a weight to each of the time-series gazing point codes and outputting a time-series weighted gazing point code;
Text information corresponding to an integrated code that integrates the time-series weighted speech code and the time-series weighted gaze point code is formed using character string information selected from character string information stored in a storage means. A multimodal speech recognition method, comprising: an eighth step of:

Images