JP2000206986A - Language information detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To input language information such as voices without necessarily depending on pronunciation, rather based on the correspondence between pronunciation and the motion of sound adjustment organs. SOLUTION: A proper processing is conducted for voice information inputted from a microphone 100 and position information of sound adjustment organ inputted by a three dimensional digitizer section 110 and the process results are recorded in a table section 120. Then, the motion of the sound adjustment organ inputted by the section 110 is inputted as time patterns of the position of each measured portion and the pattern having a highest degree of similarity to the motion information of sound adjustment organs that are recorded in the section 120, is retrieved. Then, sound information, that is made correspond to the motion of the inputted sound adjustment organ, is estimated. Thus, voice information being pronounced is obtained from the picture of the sound adjustment organs.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for inputting or recognizing linguistic information such as speech and articulator movements. More specifically, a correspondence between information such as voice and movement of the articulator is recorded at any time during normal use, and based on this correspondence, a language such as a speech is obtained from at least one of speech or movement of the articulator. The present invention relates to a device for inputting or recognizing information.

[0002]

2. Description of the Related Art A microphone is widely used as a device for inputting linguistic information such as voice as acoustic information. A method of recognizing linguistic information such as voice as text information or document information composed of a character string is widely known as voice recognition, and a part thereof has been put to practical use. In voice input and voice recognition, a voice uttered by a speaker is input as acoustic information using a microphone or the like. Inevitably, the speaker needs to speak. The acoustic information input from the microphone is output as an audio signal from the microphone regardless of whether the information is derived from speech. Therefore, in a situation where the noise is loud, there is a problem that the noise is output as an audio signal in addition to the desired audio information. Therefore, an apparatus that uses a plurality of microphones and signal processing to output ambient noise as an audio signal is disclosed in Japanese Unexamined Patent Publication No. 5-22.
392 and JP-A-2-072398.

[0003] Japanese Patent Application Laid-Open No. Hei 5-22392 discloses a portable telephone having a handset shape, a first microphone for collecting a transmission voice and outputting a transmission voice signal, and collecting a surrounding noise. And a second microphone for outputting an ambient noise signal, and an ambient noise included in the transmission voice signal output from the first microphone based on the ambient noise signal output from the second microphone. There is shown a portable radio telephone comprising a noise elimination circuit for performing arithmetic processing for removing components.

Japanese Patent Application Laid-Open No. 2-072398 discloses a method for improving a signal-to-noise ratio under noise by using two or more microphones and a filter of a neural network type for inputting signals from the microphones. It is shown.

According to these methods, it is possible to improve the signal-to-noise ratio under a predetermined condition, but it has been difficult in principle to separate the acoustic information from the utterance from the noise.

In speech recognition, there is a proposal to use not only a speech signal but also image information (Japanese Patent Application Laid-Open No. Hei 9-1997).
No. 134194). Japanese Patent Application Laid-Open No. Hei 9-134194 discloses an image recognition system that performs speech recognition based on image information acquired by a camera housed in a transmitter housing. By using such a speech recognition system to reinforce speech recognition,
While maintaining the recognition accuracy of speech recognition, speech recognition that can be applied to more people for more vocabulary becomes possible. However, this proposal does not refer to a method of constructing a reference dictionary used in image recognition, and is based on reinforcement of speech recognition of speech.
It presupposes utterance substantially. Therefore, it is difficult to avoid the influence of noise. In addition, recognition based on voice recognition or image recognition has a problem that the calculation time is relatively long.

[0007] Further, as described above, it is necessary to speak in the voice input using a microphone. However, uttering a voice in a highly public place or the like is a social problem. For example, in a shared bus or a train, use of a mobile phone is often restricted. Also, even in a quiet office or library, voice input by speech may not be appropriate. It is also assumed that utterance in situations that require confidential talk will be a challenge. In such an environment, voice input using a microphone that is supposed to speak is not appropriate.

As an input device in such a situation, a keyboard or handwritten character recognition is used. However, these input methods have a problem that a certain amount of learning is required using hands.

[0009]

SUMMARY OF THE INVENTION It is an object of the present invention to provide an input device which does not necessarily assume speech based on the correspondence between speech and motion of articulatory organs when detecting language information such as speech. As an issue. It is another object of the present invention to improve the accuracy by learning the correspondence between the utterance used in the input device and the appearance of the articulatory organ according to use and skill.

[0010]

The invention according to claim 1 is a linguistic information associating device for associating articulatory movement with linguistic information from a source other than the articulatory movement, which is generated when a speaker speaks. Articulatory organ input means for measuring the articulatory movement from at least a part of the articulatory organ and its surrounding skin to generate input data; means for inputting linguistic information other than the articulatory organ movement; And a means for associating the motion of the articulator with language information input from other than the motion of the articulator. In particular, the information input from other than the movement of the articulatory organ is defined in claim 2
Voice information as in the invention of (1). Here, it is only necessary that the information input from other than the articulator motion is associated with the articulator motion. Therefore, as described in claim 3 of the present invention, the user instructs the utterance to be uttered, and the user It is possible to recognize the movement of the articulatory organ when speaking based on the instruction. Further, as described in claim 4 of the present invention, it is also possible to recognize information consisting of phonetic characters, phonetic character strings, arbitrary symbols or symbol strings.

The linguistic information refers to information used for transmitting semantic contents, such as speech information, character information, symbols, and movements of articulators.

According to the fifth aspect, based on the correspondence between the speech information and the motion of the articulator, the information inputted from the motion of the articulator other than the motion of the articulator is based on the correspondence between the articulator and the motion of the articulator. It is also possible to acquire, and it is also possible to estimate the movement of the associated articulator from information input from other than the movement of the articulator, as described in claim 6. At this time, in order to recognize the association,
It may include the means described in claims 1 to 4, or may input information associated in advance. Of these records,
Since information input from articulator movements or articulator movements is a time pattern, in order to search for correspondence, a neural network that has learned the correspondence,
General dynamic programming or the like can be used for time pattern matching.

When detecting a sound or the like, there are two inputs, such as an acoustic input, other than the motion of the articulator, and an input by the motion of the articulator. Therefore, in a situation where it is difficult to detect only one of them, the present invention is applied. As in the seventh aspect of the invention, it is also possible to detect a voice or the like by integrating the other information. In a normal use state, it is also possible to record a dictionary at any time and perform learning, and if it is determined that one of them cannot be used, a voice can be detected from the other. In this case, either one may be selected as in the ninth aspect of the present invention, or both may be combined as in the tenth aspect of the present invention. According to the claims, the association can be recorded during normal use. At this time, as a basis for determining whether or not the detection is difficult, a certainty factor can be calculated and used as a basis as in the inventions of claims 10 and 11. In a situation where the utterance is inappropriate, whether the most important utterance of the utterance certainty level is voiced or silent is mentioned. According to the twelfth aspect of the present invention, the certainty factor is obtained by recognizing whether the utterance is voiced or silent.

Further, the movement of the organ of the speaker and the semantic information such as gestures may be associated with each other as in the invention of claim 8.

The voice information input from other than the motion of the articulator may include a voice signal input by a microphone or the like, a time pattern obtained by converting the voice signal into a signal intensity in an appropriate frequency band, and the like. Although the result of voice recognition can be used, voice information can be input even if phonetic characters are explicitly input from a keyboard or the like.

The articulator form input means for generating input data by measuring the movement of the articulator generated when the speaker speaks from at least a part of the articulator and the surrounding skin is performed by using a sensor in contact with the skin. Can also be entered. Further, as described in claim 13 of the present invention, an appropriate sensor for the appearance of the outer skin, for example, a sensor for optically inputting a three-dimensional shape may be used.
It is also possible to detect the movement by taking an image of the outer skin as shown in FIG. As a method of detecting motion from an image,
An optical flow method is known.

The voice information input from other than the articulator movement and the input data obtained by measuring the movement of the articulator generated when the speaker speaks from at least a part of the articulator and its surrounding skin are both time data. Can be considered as a time pattern that varies depending on the time. Therefore, as a method of associating the voice information input from other than the articulator movement with the movement of the articulator during speech, a method of recording these in a table as a time pattern can be considered. As described above, by using a neural network such as a hierarchical neural network or a recurrent neural network, it is possible to perform correspondence by learning, for example, a time pattern of exercise as an input signal and a corresponding phonogram as a teacher signal. Similarly, correspondence can be performed by learning phonograms as input signals and time patterns of exercise as teacher signals.

The data of the input signal and the teacher signal, which are the basis of the learning, are obtained by recording the voice information input from other than the movement of the articulator and the movement of the articulator during speech as in claim 17. Obtainable. Here, the data to be recorded is recorded and updated at any time or based on an instruction based on the input voice information and the movement of the articulator during speech as described in claim 18. However, it is also possible to calculate the certainty factor of each input and automatically control the recording according to the certainty factor. For example, it is possible to perform control such as updating the record only when each certainty factor exceeds 90%. Further, in the case where one of the inputted certainty factors is equal to or less than a predetermined value in the process of detecting the voice, the other input and the other characteristic corresponding to the input are detected. The detection result is output based on the information.If both confidences are equal to or greater than a predetermined value, the detection means is provided with means for controlling recording and updating in accordance with both the confidences obtained. It is also possible to output. According to the present invention, it is possible to improve the recognition rate while recording the correspondence of the input and output with a high degree of certainty during normal use. Similarly, the present invention can be applied to speech recognition.

The above invention can be applied to a telephone device as described in claim 23. The telephone device includes, in addition to a normal stationary telephone, a portable wireless telephone device and a videophone. By applying to such a telephone device, it is possible to use in a situation where speech is difficult or in a situation where noise is large. In addition, video cameras and audiovisual devices typified by video tape recorders that essentially handle audio information and image information can easily acquire audio signals and image signals. There are provided means for inputting voice information input from a user, and means for inputting images of the articulator and its surrounding skin, which indicate the movement of the articulator generated when the speaker speaks.

In particular, in a videophone, the movement of the articulator can be associated from the audio information, so that the transmission of the movement of the articulator can be omitted or reduced.

Further, when the present invention is applied to a device for inputting a document composed of characters or character strings, it is possible to improve the certainty of input. In addition, movement of articulatory organs different from normal speech, for example, remarkably bilaterally asymmetrical movement,
It can be recognized as a signal different from the utterance.
By allocating such exercises to appropriate gestures, it is possible to configure a device that inputs gestures that cannot be achieved by voice recognition alone.

Although the above invention has been described with respect to each means, some or all of these means may be implemented as a computer program. Further, it is possible to record these computer programs and associations on an appropriate medium.

[0023]

According to the present invention, uttered acoustic information and motion of articulatory organs are integrated in order to input speech information or transmit speech.

The uttered sound information correlates sound information input by voice input means such as a microphone with motion information of articulators input by a three-dimensional digitizer or an image input device using a table or a neural network.

According to this association, if there is a problem in one of the inputs, the input is made from the other input. If it is assumed that any input is correct, the information is used as information for association.

As a device for associating voice information with articulatory movement, the information input by means for inputting voice information and the means for inputting the motion of the articulator during speech are provided. An appropriate process is performed as needed, and the result is recorded in a table, or learning of a neural network is performed as in the inventions of claims 15 and 16.

Based on this association, voice information is obtained from the motion of the articulator, as in the invention of claim 2, or the motion of the articulator is obtained based on the voice, as in the invention of claim 3. presume.

At this time, the voice information may be recorded (recorded) as it is, or may be a time pattern of a frequency spectrum, or may be subjected to voice recognition and the resulting phonetic symbols, characters or character information, Alternatively, it may be recognized as document information composed of a character string.

The motion of the articulator may be recorded (recorded) as it is in the case of an image, or may be recorded as a shape recognition or its motion using an optical flow method or the like. However, when recording in a table or neural network and indexing the table based on the similarity of exercise, or when assuming input from a neural network, it is sometimes desirable to digitize both voice information and exercise information. is there. For example, in the input exercise information,
When searching for the motion information with the highest similarity in the table by the dynamic programming method, it is difficult to process with the recorded information, but if it is recorded as the time pattern of the position as the motion information of the predetermined position, it can be easily performed. You can search.

[0030]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The features of the present invention will be specifically described below based on embodiments with reference to the drawings. Embodiment 1 FIG. 1 shows Embodiment 1 of the present invention. The first embodiment is realized as a linguistic information associating device for associating audio information with motion of articulators.

In FIG. 1, the language information associating device
It includes a microphone 100, a three-dimensional digitizer 110, a table 120, and a spectrum information calculator 130. The audio signal input by the microphone 100 and the position information of the articulator including the face synchronized with the audio signal input from the three-dimensional digitizer 110 that contacts the articulator are recorded in the table unit 120. At this time, the voice information is obtained by the spectrum information calculation unit 130.
It is recorded as a time pattern of the spectrum intensity for each appropriate frequency band. In addition, the position information of the articulator including the face includes three or more trajectories of one or more appropriate points on the articulator (for example, both ends of lips, centers of upper and lower lips, center position of throat, and center of cheek). Input by the dimensional digitizer 110,
It is recorded as a time pattern of the position of each measurement site.

The table 120 allows the input voice information to correspond to the input motion of the articulator during speech.

The table section 120 may be appropriately rearranged so that the index is easy. In this embodiment, a three-dimensional digitizer is used as a means for inputting articulatory movement. However, any other method may be used as long as it can input articulatory movement.

FIG. 2 shows a second embodiment of the present invention. The second embodiment is realized as an audio information estimating apparatus for estimating audio information from the movement of articulators. In FIG. 2, the portions corresponding to those in FIG.

In FIG. 2, similarly to the first embodiment, the voice information input by the microphone 100 and the position information of the articulator input by the three-dimensional digitizer 110 are appropriately processed, and the table 120 To record.

Next, the motion of the articulator entered by the three-dimensional digitizer unit 110 is input as a time pattern of the position of each measurement site, and recorded in the table unit 120 in the same manner as when recorded in the table unit 120. A search is made for the motion information of a given articulator and the one with the highest similarity. As an example of this method, a dynamic programming method widely used for recognizing a time pattern may be used.

As described above, the table section 120 is searched, and the acoustic information associated with the input motion of the articulator is estimated.

According to the present embodiment, it is possible to acquire speech information of speech from an image of an articulator without performing speech recognition. For example, when a human hears directly as acoustic information or uses it as an input for advanced speech recognition, there is a slight error due to consideration of information such as context and other natural language information. Can be corrected. For example, in Japanese, if a vowel is almost accurate, it is often possible to understand the contents of a sentence by taking into account information such as context, even if there are some errors in consonants.

In this embodiment, an example is shown in which the means up to recording in the table section 120 and the means from the index of the table section 120 are implemented by the same apparatus. Can be transmitted by communication or the like, the present invention can be implemented without being implemented in the same device.

Third Embodiment FIG. 3 shows a third embodiment of the present invention. The third embodiment is realized as an articulator motion estimation device that estimates the motion of the articulator based on speech. In FIG. 3, the portions corresponding to FIG. 1 are denoted by the corresponding reference numerals.

As in the first embodiment, the voice information input by the microphone 100 and the three-dimensional digitizer 11
Processing is appropriately performed on the position information of the articulatory organ input by 0, and is recorded in the table unit 120.

Next, in the same manner as when recording on the table unit 120, the audio signal input by the microphone 100 is input by the spectrum information calculation unit 130 as an appropriate time pattern of the spectrum intensity for each frequency band. The voice information recorded in the section 120 and the one having the highest similarity are searched. As an example of this method, a dynamic programming method widely used for recognizing a time pattern may be used.

As described above, the table section 120 is searched to estimate the movement of the articulator corresponding to the input speech information.

According to this embodiment, based on the voice easily,
The articulatory movement can be estimated. The unit up to the recording in the table unit 120 and the unit after indexing the table unit 120 may be in the same device,
The information recorded in the table unit 120 may be transmitted between the apparatuses by installing the apparatuses in different apparatuses.

Fourth Embodiment FIG. 4 shows a fourth embodiment of the present invention. Embodiment 4 is implemented as a videophone device. In FIG. 4 as well, portions corresponding to FIG. 1 are denoted by corresponding reference numerals.

In FIG. 4, as in the third embodiment, the motion of the articulator associated with the input speech information is estimated.

The associating table 120 and the input audio signal are transmitted to the receiving side via the transmission path 220.
The receiving side estimates the exercise information corresponding to the voice from the transmitted table 120, and estimates the exercise information. An image of the face surface is allocated to the estimated motion of the articulator by the texture mapping unit 200. The image is displayed on the display unit 2
Indicated by 10.

According to the present embodiment, regarding the articulatory organ,
It is possible to transmit the movement of the articulator in the face image without transmitting the movement at any time.

Embodiment 5 FIG. 5 (a) shows Embodiment 5 of the present invention. Example 5 shows an apparatus for associating audio information with motion of articulators. In FIG. 5A, the same reference numerals are given to the portions corresponding to FIG.

In FIG. 5A, the microphone 100
In the table unit 120, the audio signal input by the above and the position information of the articulator including the face synchronized with the audio signal input from the three-dimensional digitizer unit 110 that contacts the articulator are recorded. At this time, the voice information is output by the phoneme recognition unit 131.
Record as vowel and consonant time patterns. In addition, the position information of the articulator including the face includes three or more trajectories of one or more appropriate points on the articulator (for example, both ends of lips, centers of upper and lower lips, center position of throat, and center of cheek). The data is input by the dimensional digitizer unit 110 and recorded as a time pattern of the position of each measurement site.

The table unit 120 makes it possible to associate the input phoneme information with the input motion of the articulator during speech.

The table section 120 may be appropriately rearranged so that the index is easy. In this embodiment, a three-dimensional digitizer is used as a means for inputting articulatory movement. However, any other method may be used as long as it can input articulatory movement.

As shown in FIG. 5B, a voice recognition unit 132 is used in place of the phoneme recognition unit 131 in this embodiment, and voice information is recognized as text information or document information composed of a character string. Is also possible.

Embodiment 6 FIG. 6 shows Embodiment 6 of the present invention. In this embodiment, the present invention is applied to a portable radio telephone device. This embodiment is described in JP-A-9-13419.
No. 4 discloses a handset in which the present invention is mounted. Also in FIG. 6, the portions corresponding to FIG.

The portable wireless telephone device allows a wider range of situations in which a telephone call can be made than a normal wired telephone, and, for example, a telephone call is possible in principle even in public transportation. But,
It is generally considered that in such a situation, utterance toward the telephone is annoying. In the present embodiment, use in such a situation is assumed. Also,
This embodiment can be applied not only to a portable radio telephone device but also to other telephone devices.

In FIG. 6, when speech is possible, the voice signal input and transmitted by the microphone 100 and the voice signal input from the video camera 111 of the portable wireless telephone device that comes into contact with the articulator are converted to the voice signal. The position information of the articulator including the synchronized face is recorded in the table unit 120. At this time, the voice information is converted by the voice conversion unit 133 so as to be a parameter when the voice similar to the input voice is synthesized by the voice synthesis unit 140, and recorded in the table unit 120. In this process, speech recognition may be performed, but it is also possible to calculate a parameter at a level at which sound information that can be heard almost equally can be synthesized without necessarily performing recognition.

The table section 120 makes it possible to associate the speech synthesis parameters with the movements of the articulator during speech.

The table section 120 may be appropriately rearranged so that the index is easy. Further, in the present embodiment, the video camera 111 of the portable wireless telephone device that contacts the articulator is used as a means for inputting the motion of the articulator, but if the means can input the motion of the articulator, image input, Other methods such as distance measurement by ultrasonic waves may be used.

In a situation where there is a problem in utterance, it is used according to the following procedure based on a user's instruction.

First, the motion of the articulator recorded in the table unit 120 closest to the position information of the articulator including the face input from the video camera 111 of the portable radiotelephone device that comes into contact with the articulator is calculated by a dynamic programming method. Search for. The voice synthesis parameters corresponding to this motion are indexed, and voice is synthesized by the voice synthesis unit 140.
The synthesized voice is transmitted by the switching unit 141 instead of the output from the microphone 100.

In the present embodiment, recording on the table, indexing and synthesizing of the table are switched based on a user's instruction. However, the above switching may be automatically performed according to the presence or absence of an input from the microphone 100. It is possible. It is also possible to add means for detecting noise or the like, and to switch according to the level of the noise. When switching is performed automatically, for example, in a situation where the input from the microphone 100 is interrupted, assuming that a part is synthesized according to the present invention, even at a level at which a vowel can be understood at most, communication is practically impossible. Quality can be greatly improved.

Seventh Embodiment FIG. 7 shows a seventh embodiment of the present invention. In the seventh embodiment, the present invention is applied to a speech recognition device. Also in FIG. 7, the portions corresponding to those in FIG.

With the development of speech recognition technology in recent years, a speech recognition technology of a practical level has been put to practical use depending on the application. However, since the current speech recognition technology is based on the premise that speech is generated, there is a case where use in a public transportation, a quiet office or a library is hindered. In addition, there is a problem that there is a problem in a method of inputting only voice when noise is loud in a vehicle or the like.
In the present embodiment, speech recognition in such a situation is assumed.

In a situation in which voice recognition can be performed only by utterance, a video camera 11 that captures a voice signal transmitted by the microphone 100 and transmits the voice and an articulatory organ.
The position information obtained from the image of the articulator including the face synchronized with the audio signal input from step 1 is recorded in the table unit 120. At this time, the voice information is recognized by the voice recognition unit 132 and recorded in the table unit 120. In addition, as a method of acquiring the movement of the articulator from the video image, an optical flow method can be used.

Here, the determination as to whether or not the situation is such that voice recognition can be made only by utterance may be specified by the user,
At the time of speech recognition, a certainty factor may be calculated, and a determination may be made based on whether the certainty factor is equal to or greater than a predetermined value.

Next, at the time of voice recognition, recognition is performed according to the following procedure.

The voice signal input and transmitted by the microphone 100 and the position information of the articulator including the face synchronized with the audio signal input from the video image of the articulator are simultaneously input. For this input and recognition, the means used when recording in the table unit 120 may be used, or different means may be used if the results are equivalent. On this occasion,
Recognition is also performed together with the certainty factor of the input voice information. For example, by detecting noise or the like, a probability that the recognition result is correctly recognized (not erroneous recognition) is calculated.

If the probability is equal to or greater than a predetermined value, the result of speech recognition is adopted as a recognition result. Otherwise, the result of recognizing the movement of the articulator is adopted as the recognition result.

In the above description, the certainty factor of only speech recognition is used. However, the certainty factor is also calculated for the result of recognition of the articulatory movement, and a recognition result with a high certainty factor may be employed, or the total confidence factor may be used. A recognition result with a high degree may be adopted. Alternatively, only the certainty of the articulator may be obtained, and selection or synthesis may be controlled based on the obtained confidence.

Also, the recognition result of the consonant and the vowel may be synthesized by using the result of speech recognition as the consonant and using only the vowel having a high degree of certainty. Thus, various combinations and methods can be used for selection and synthesis.

[Eighth Embodiment] FIG. 8 shows an eighth embodiment of the present invention. This embodiment 8 shows an apparatus for associating character information with movement of articulatory organs. Also in FIG. 8, the portions corresponding to those in FIG.

The character information input by the microphone 100 and subjected to voice recognition and the position information of the articulator including the face synchronized with the audio signal input from the three-dimensional digitizer unit 110 that contacts the articulator are acquired. At this time, the character information uttered by the user may be input using a keyboard or the like without inputting and recognizing the voice, or the character information to be uttered may be presented to the user by an appropriate instruction device, and the utterance may be performed. May be.
At this time, the utterance is performed only by the movement of the articulatory organ, and the utterance does not have to be made or the utterance may be made.

Using the character information associated with the above and the motion information of the articulator, the neural network 1
Learn 21. At this time, the motion information of the articulator is used as a teacher signal, and the character information is used as the neural network 121.
Give to. Then, using the recognition rate as an energy function, the neural network 1
Learning is performed by adjusting the 21 coefficients. Here, the recognition rate is used as the energy function, but an appropriate ideal output corresponding to the character information may be used as the teacher signal, and the Euclidean distance between the teacher signal and the output of the neural network 121 may be used as the energy function.

For recognition of such a time pattern, a general hierarchical neural network may be used.
A recurrent neural network may be used.

As described above, learning is repeated until the energy function becomes equal to or less than the predetermined value or until the predetermined number of times.

At the time when the learning is completed, the speech recognition is performed in the following procedure.

Three-dimensional digitizer 1 that contacts the articulator
The position information of the articulator including the face synchronized with the audio signal input from the device 10 is acquired. The position information or the time pattern of the position information is input to the neural network 121. If necessary, the neural network 12
By converting the output of No. 1 into character information, a recognition result can be calculated.

Recognition may be performed on a character basis or on a document basis consisting of character strings. In the case of performing recognition in units of documents composed of character strings, the recognition rate can be improved in consideration of the appearance frequency, context, and the like.

Although the neural network 121 is used in place of the table section 120 here, other means,
For example, the present invention may be used for “a means for selecting or synthesizing an output from at least one of input voice information and associated voice information” in claim 4 or a means for recognizing the certainty factor in claim 5. May be used. in this case,
If learning is not required, it is not necessary to perform learning.

Also, by assigning and recognizing the movement of the articulatory organ that does not correspond to a document composed of a character or a character string, for example, a left-right asymmetrical movement, to “yes”, “no”, pointing, etc. Claim 22
It is also possible to recognize a gesture as shown in FIG.

Ninth Embodiment FIG. 9 shows a ninth embodiment of the present invention. The ninth embodiment is realized as a device for associating character information with the movement of the articulator. This embodiment is shown in FIG.
In this case, the neural network 121 is used in the same manner as described above. In FIG. 9 as well, portions corresponding to those in FIG.

As in the eighth embodiment, the microphone 100
Then, using the three-dimensional digitizer unit 110 that comes into contact with the articulator, the result of performing appropriate processing is recorded in the recording unit 170 as needed or based on an instruction. Next, learning of the neural network is performed based on this record as in the eighth embodiment. The recognition is the same as in the eighth embodiment.

According to this embodiment, learning can be performed at any time during use, and the recognition rate can be improved with use.

Further, by using the presentation means 160 for presenting the recognition result to the user, the user can train the movement of the articulator so as to increase the recognition rate. According to this embodiment, it is also possible to apply to speech training for a hearing-impaired person. Here, the presentation may be presented visually or may be presented to a strong sound enemy.

[Embodiment 10] FIG. 10 shows an embodiment 10 of the present invention. The tenth embodiment is realized as a speech recognition device. Also in FIG. 10, portions corresponding to those in FIG.

Microphone 100 and Video Camera 1
11 shows an example of performing high-reliability voice recognition from a voice and an articulator image in a voice and a video image input or a voice and video image recorded on a video tape recorder 180 and reproduced. Show.

The speech recognition unit 132 recognizes the speech from the input or reproduced speech. In addition, confidence 1
91 is calculated.

Further, the motion of the articulator is recognized by the optical flow analysis unit 150 from the image of the articulator in the reproduced image. Together with the recognition result, the confidence 192
Is calculated.

Based on these two sets of recognition results and certainty factors, the recognition result determination unit 190 selects and outputs the recognition result with the higher certainty factor.

[0090]

As described above, according to the present invention, it is possible to provide an input method which does not require speech when inputting audio information based on the correspondence between the audio information and the appearance of the articulator. . In addition, it is possible to improve the accuracy by learning the correspondence between the utterance and the appearance of the articulatory organ according to the use and skill.

[Brief description of the drawings]

FIG. 1 is a diagram showing a first embodiment of the present invention.

FIG. 2 is a diagram showing a second embodiment of the present invention.

FIG. 3 is a diagram showing a third embodiment of the present invention.

FIG. 4 is a diagram showing a fourth embodiment of the present invention.

FIG. 5 is a diagram showing an example of Embodiment 5 of the present invention.

FIG. 6 is a view showing a sixth embodiment of the present invention.

FIG. 7 is a diagram showing a seventh embodiment of the present invention.

FIG. 8 is a diagram showing an eighth embodiment of the present invention.

FIG. 9 is a view showing a ninth embodiment of the present invention.

FIG. 10 is a diagram showing a tenth embodiment of the present invention.

[Explanation of symbols]

 100 Microphone 110 3D digitizer 111 Video camera 120 Table 121 Neural network 130 Spectrum information calculator 131 Phoneme recognizer 132 Voice recognizer 133 Voice converter 140 Voice synthesizer 141 Switching Unit 150: Optical flow analysis unit 160: Presentation means 180: Video tape recorder 190: Recognition result determination unit 191: Confidence of speech recognition 192: Confidence of optical flow analysis unit 200: Texture mapping unit 210: Display unit 220: Transmission Road and

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Shin Takeuchi 430 Sakai Nakagami-cho, Ashigara-gun, Kanagawa Green Tech Nakai Fuji Xerox Co., Ltd. F-term (reference) 5D015 CC00 LL07 9A001 BB04 HZ05 KZ20

Claims (23)

[Claims] 1. A linguistic information associating device for associating articulatory movement with linguistic information other than the articulatory movement, wherein the articulatory movement generated when a speaker speaks is determined by the articulator and its surroundings. Articulatory morphology input means for generating input data by measuring from at least a portion of the hull; means for inputting linguistic information other than motion of the articulator; motion of the articulator during speech; Means for associating linguistic information input from other than exercise. 2. The linguistic information associating device according to claim 1, wherein the linguistic information other than the movement of the articulator is speech information. 3. A linguistic information associating device for associating articulatory movements with linguistic information other than articulatory movements, comprising: means for instructing a speaker to speak and the contents of the speech; and the speaker speaking. An articulator form input means for generating input data by measuring the movement of the articulator that occurs at least from at least a part of the articulator and its surrounding skin; and a motion other than the motion of the articulator during the speech and the motion of the articulator Means for associating linguistic information input from the language information. 4. A linguistic information associating apparatus for associating articulatory movement with linguistic information other than the articulatory movement, wherein the articulatory movement generated when a speaker speaks is determined by using the articulatory organ and its surroundings. Articulatory form input means for generating input data by measuring from at least a part of the hull, means for inputting information comprising phonetic characters, phonetic character strings, arbitrary symbols or symbol strings, and movement of the articulatory organs A language information associating device, comprising: means for associating information input from other sources with the movement of the articulator during speech. 5. A linguistic information estimating apparatus for estimating linguistic information other than the articulator movement from the articulator movement, the linguistic information estimating apparatus comprising: A matching recognizing means for recognizing a correspondence between the articulators; and an articulator organ shape input for generating input data by measuring motion of the articulators generated when the speaker speaks from at least a part of the articulators and the surrounding skin. Means for acquiring corresponding linguistic information from the movement of the articulator during speech based on the correspondence recognized by the correspondence recognizing means. 6. An articulator motion estimation device for estimating articulator motion based on information other than articulator motion, comprising: articulator motion during speech; and information input from other than articulator motion. A correspondence recognizing means for recognizing the correspondence between: a means for inputting linguistic information other than the movement of the articulator; and a means other than the movement of the articulator based on the correspondence recognized by the correspondence recognizing means. An articulator motion estimation device comprising means for estimating articulator motion during speech. 7. A linguistic information detecting apparatus for detecting linguistic information such as a voice, wherein the linguistic information detecting apparatus recognizes a correspondence between a motion of an articulator during speech and linguistic information input from a source other than the motion of the articulator. Attachment recognition means; language information input means for inputting linguistic information other than movement of the articulator; and movement of the articulator generated when the speaker speaks is measured and input from at least a part of the articulator and its surrounding skin. Articulatory form input means for generating data; linguistic information acquiring means for acquiring corresponding linguistic information from the movement of the articulator during speech based on the association recognized by the association recognizing means; The linguistic information is obtained based on the linguistic information obtained from the movement of the articulator by the linguistic information obtaining means, and the linguistic information inputted from other than the movement of the articulator by the linguistic information input means. Language information detecting apparatus characterized by having language information detecting means for detecting. 8. The apparatus further comprises means for recognizing a correspondence between movement of a speaker's organ and semantic information such as a gesture,
The linguistic information detecting device according to claim 7, wherein the semantic information is detected using the association. 9. The linguistic information detecting means selects linguistic information acquired from the movement of the articulator by the linguistic information acquiring means, and linguistic information inputted from other than the articulator movement by the linguistic information input means. The linguistic information obtained from the movement of the articulator by the selecting means and the linguistic information obtaining means, and the linguistic information inputted from other than the movement of the articulator by the linguistic information input means. The linguistic information detecting device according to claim 7, further comprising at least one of a synthesizing unit that generates a detection result. 10. The at least one of a means for recognizing certainty of linguistic information associated with articulatory movement and a means for recognizing certainty of linguistic information input from a source other than articulatory movement. Linguistic information obtained from the movement of the articulator, depending on one or both confidence factors,
10. The linguistic information detecting device according to claim 9, wherein one of the linguistic information input from other than the motion of the articulator is selected. 11. The synthesizing means includes at least one of means for recognizing certainty of linguistic information associated with articulatory movement, and means for recognizing certainty of linguistic information input from a source other than articulatory movement. Linguistic information obtained from the movement of the articulator, depending on one or both confidence factors,
11. The linguistic information detecting device according to claim 9, wherein linguistic information input from other than the movement of the articulator is synthesized. 12. The language according to claim 10, wherein the means for recognizing the certainty factor of the linguistic information input from a source other than the movement of the articulator includes a means for recognizing whether the utterance is sound or silence. Information detection device. 13. The linguistic information detecting apparatus according to claim 7, wherein said articulator organ shape input means includes means for inputting a shape of a skin. 14. The linguistic information detecting device according to claim 13, wherein the means for inputting the shape of the outer skin includes means for inputting an image of the outer skin. 15. The linguistic information detecting device according to claim 7, wherein the association recognizing unit, the linguistic information input unit, and the linguistic information acquiring unit include a neural network. 16. The linguistic information detecting device according to claim 15, wherein said association recognizing means performs learning of a neural network. 17. The linguistic information detecting apparatus according to claim 7, further comprising means for recording the movement of the articulator during speech and information input from a source other than the movement of the articulator. 18. The language according to claim 17, wherein the correspondence is recorded and updated at any time or based on an instruction based on movement of the articulator during speech and information input from other than movement of the articulator. Information detection device. 19. A system comprising: means for calculating the certainty factor of the movement of the articulator during speech; and means for calculating the certainty factor of the information input from other than the movement of the articulator. 19. The language information detecting device according to claim 18, further comprising means for controlling recording and updating in accordance with the language information. 20. Means for calculating the certainty of the movement of the articulator during speech, and means for calculating the certainty of information input from other than the movement of the articulator, wherein one of the inputted certainties is determined. If the value is equal to or less than a predetermined value, a detection result is output based on the other input and the other characteristic corresponding to the input, and when both confidences are equal to or more than a predetermined value,
19. The linguistic information detection device according to claim 7, wherein the linguistic information detection device outputs a detection result at the same time as updating the correspondence according to the obtained certainty factors. 21. In the case of performing voice recognition, a recognition result based on the motion information of the articulator, a result of recognizing information input from a source other than the motion of the articulator, or a result of selecting or synthesizing the information is used. 21. The language information detecting device according to claim 7, further comprising means for presenting to a person. 22. A method according to claim 1, further comprising: inputting information other than the articulator movement associated with the articulator movement, and the articulator movement during speech. A medium that records the correspondence of 23. The linguistic information associating device according to claim 1, 3, 3 or 4, the linguistic information estimating device according to claim 5, the articulator movement estimating device according to claim 6, or A telephone device comprising the language information detecting device according to claim 7.