JP4189744B2 - Voiceless communication system - Google Patents

Description

  The present invention relates to a voiceless communication system, and more particularly, to a voiceless communication system that enables a telephone call without producing a voice.

  As conventional silent voice communication means, there are known a voice processing device and a telephone device comprising a transmitter for recognizing a voice from a lip shape and transmitting a signal, and a receiver for receiving the signal and decoding it into a voice. (See Patent Document 1, FIGS. 6 and 11, paragraph 0039).

  FIG. 9 is a diagram illustrating a configuration of the apparatus described in Patent Document 1. In FIG. As shown in FIG. 9 (a), the transmitter of this apparatus inputs the lip shape image data and outputs the speech synthesis original data corresponding to the lip shape image data, and the voice A voice synthesizer 2a, a voice coder 3a, and a transmitter 4a that perform voice synthesis, encoding, and transmission from the original data for synthesis, respectively, and a receiver side that has a receiver 5a and a voice decoder 6a It tries to be able to make a call without taking it out. As shown in FIG. 9B, the lip shape recognizing unit 91 stores an input unit 1b that captures the shape of the vicinity of the lip and outputs data, and original data for speech synthesis corresponding to the image data is stored as a database 5b. The memory unit 4b and the control unit 2b that outputs original data for speech synthesis corresponding to the image data from the input unit 1b to the data output unit 3b.

Japanese Patent Laid-Open No. 10-240283

  Conventional speech processing devices and telephone devices are capable of generating speech data to be spoken by the lips, but simply generate original data for speech synthesis corresponding to image data based on the database. However, there is no disclosure of a specific logical recognition process for correctly generating a voice intended by the user.

(the purpose)
An object of the present invention is to provide a novel silent communication system that recognizes the shape of a user's mouth and outputs a correct voice intended according to the shape.
An object of the present invention is to provide a voiceless voice communication system including a user's confirmation operation for recognizing a mouth shape.
Another object of the present invention is to provide a voiceless communication system using a mask for photographing the shape of the mouth.

  The present invention provides a novel means for performing correct voice utterance by identifying a mouth shape when a user speaks and synthesizing the voice in order to enable a call in a situation where voice utterance is difficult. The voiceless call system of the present invention includes an image photographing unit for photographing the mouth, an image analyzing unit for generating character data from the photographed image data, a character string converting unit for generating character string data from the character data, A voice synthesis output unit that synthesizes voice data from the character string data, a voice output unit (for example, 12 in FIG. 1) that converts the voice data into voice, and whether or not to output the character string data to an external output unit A recognition result determination switch (for example, 9 in FIG. 1), and a basic mouth shape image database in which the relationship between the shape of the mouth portion of the image data and the character data is used as a database. Character data is generated from the shape of the mouth of the photographed image with reference to the basic mouth image database.

  The image analysis unit thins out the subsequent character data of the continuous character data when a certain number of the same character data continues, and outputs the thinned character data to the character string conversion unit. The character string converter is controlled to output as data.

  Furthermore, a mask for covering the mouth part and photographing the mouth part is provided, and at least the illumination part for illuminating the mouth part and the image photographing part for photographing the mouth part are arranged inside the mask.

More specifically, the portion of the user's mouth illuminated by the illumination unit 3 is photographed by the image photographing unit 1 at time intervals set by the timer unit 2, and the result is notified to the image analyzing unit 4. The image analysis unit 4 refers to the basic mouth image database 5 to determine what character the mouth shape is similar to when the character is emitted, and outputs the result to the character string conversion unit 6 as character data. The character string conversion unit 6 compares the character string data including a plurality of character data with the contents of the vocabulary database 7, searches a plurality of vocabularies close to the character string data, and searches the vocabulary in the lexical order having a high degree of matching from the vocabulary. Or a plurality of character string data are output as candidates to the speech synthesis output unit 10. The voice synthesis output unit 10 converts the character string data into voice data and outputs the voice data to the earphone 12. The user determines whether or not the user intended by the audible sound from the earphone 12, and operates the recognition result determination switch 13 that can be output as recognition OK or recognition NG, respectively, thereby respectively outputting the audio data to the external output unit 11 Or the audio data of the next candidate is output to the external output unit 11. In the case of recognition OK, the speech synthesis output unit 10 outputs speech data to the external output unit 11, otherwise, the speech utterance is realized by repeating the process of outputting the next candidate from the earphone 12. (Figure 1 )

  According to the present invention, voice synthesis is realized by sequentially converting the shape of the mouth into character data, the character data into character string data, and the character string data into voice data, and the user needs to speak at all. Therefore, the content of the utterance can be determined by the movement of the mouth and the operation of the hand, so that it is possible to communicate with the other party by outputting the correct content that the user wants to utter to the speaker or the microphone terminal of the telephone. This is extremely effective in environments where speech cannot be spoken, such as in trains, or inability to speak due to physical problems.

  In addition, by using a switch that controls the recognition start timing in relation to the movement of the mouth and a switch for determining the recognition result, it is possible to reliably synthesize correct voice data intended by the user. . In addition, by changing the mouth so that a certain number of the same character data continues, such as when utterance is stopped for a certain time, the character string conversion is started as a character string delimiter, so that the current point in time without switching It is possible to generate character string data up to and convert it into voice data and automatically utter it.

  In addition, by using a mask that includes an illumination unit and an image capturing unit on the inside and covers the user's mouth, the brightness of the location used by the user is not affected, and the user wants to speak It is possible to speak without being understood by users other than those who are in

(Description of configuration)
FIG. 1 is a diagram showing an embodiment of a voiceless communication system according to the present invention. The silent communication system of the present embodiment includes an image capturing unit 1 that captures the vicinity (mouth) of a user's mouth and outputs image data (images), and a shooting operation of the image capturing unit 1 at regular time intervals. A timer unit 2 for outputting a signal to be performed, an illumination unit 3 for illuminating a user's mouth brightly, an image analysis unit 4 for analyzing an image photographed by the image photographing unit 1, and a user uttering a specific character A basic mouth image database 5 that records the relationship between the image of what mouth shape is, that is, the specific character and the image of the user's mouth shape corresponding to the specific character, and image analysis A character string conversion unit 6 that outputs a character string estimated from the result of the unit 4, a vocabulary database 7 that is vocabulary data when the character string conversion unit 6 generates a character string, and a timing for converting a character into a character string From the user to the character string converter 6 A recognition start notification switch 9 for notification, a speech synthesis output unit 10 for outputting a character string as speech, an external output unit 11 for outputting speech to the outside, and a user listening to the output of the speech synthesis unit 10 And a recognition result determination switch 13 for determining whether or not the recognition result is a character string intended by the user from the output of the speech synthesizer 10 and for notifying the character string conversion unit 6 by the user. ing.

The configuration and function of each part will be described.
The image photographing unit 1 is a photographing device for photographing a user's mouth shape as a still image. The timer unit 2 is a timer capable of setting the time interval for outputting a timing signal for photographing the shape of the user's mouth at regular time intervals. A shooting instruction signal is sent from the timer unit 2 to the image photographing unit 1 at regular intervals, and the image photographing unit 1 photographs the user's mouth according to the instruction signal. The illumination unit 3 brightly illuminates the mouth portion of the user to be imaged by the image capturing unit 1 and enables a clear image to be captured.

  The image analysis unit 4 analyzes the image captured by the image capturing unit 1, identifies the closest sound when the normal user utters from the shape of the user's mouth of the image, and character data corresponding to the sound (Character) is output. The basic mouth image database 5 is a database of the relationship between the sound normally uttered by the user and the image data of the mouth shape at that time, and the image analysis unit 4 identifies the sound from the mouth shape of the user of the image. Referenced when doing.

  The character string conversion unit 6 collects a plurality of characters output from the image analysis unit 4, combines the characters uttered by the user, and specifies character string candidates to be output with reference to the vocabulary database 7. It has a function. Here, the vocabulary database 7 is a vocabulary group referred to when the character string conversion unit 6 specifies a character string to be uttered by the user, and the character string conversion unit 6 attempts to utter the user from the vocabulary database 7. Find a string and identify the string.

  The recognition start notification switch 9 is a switch that is manually operated by the user. When the user presses the recognition start notification switch 9, characters that have been output from the image analysis unit 4 to the character string conversion unit 6 so far are displayed as characters. A function for notifying conversion to a column is provided.

  The voice synthesis output unit 10 has a function of outputting the character string specified by the character string conversion unit 6 as a voice from the external output unit 11 or the earphone 12. The earphone 12 outputs the output of the voice synthesis output unit 10 to the user as a voice, and allows the user to determine whether or not to output the voice output to the external output unit 11. 11 has a function of outputting an audio output to an audio input unit such as a mobile phone.

  Based on the sound output from the earphone 12 regarding the character string specifying result in the character string conversion unit 6, the recognition result determination switch 13 determines whether the user is good or bad and outputs the sound output to the external output unit 11. Or a function of selecting whether to discard the character string and instruct the character string conversion unit 6 to specify a new character string that is the next candidate. When the instruction from the recognition result determination switch 13 is recognition OK, the voice output is switched to the external output unit 11 and provided. If the instruction from the recognition result determination switch 13 is recognition NG, the sound output is discarded.

  FIG. 2 is a diagram showing the image photographing means of the present embodiment. From a mask part A1 that can be worn on the user's mouth, an earphone part A2 that can be worn on the user's ear, a switch part A4 that can be held and operated by the hand, and an external output part A3 that can be connected to an external device or the like. Composed. In this example, the mask unit A1 can be provided with only the image capturing unit 1 shown in FIG. 1, or the image capturing unit 1 through the speech synthesis output unit 10, and the switch unit A4 can be notified of the recognition start shown in FIG. A switch 9 and a recognition result determination switch 13 are provided. The user wears the mask part A1 on the mouth part, puts the earphone part A2 on the ear, moves the mouth without making a voice, confirms the voice of the recognized character string by the earphone part A2, and switches the switch part A4. Only the correct speech recognition result by the mouth is output from the speech synthesis output to the external output unit 11.

(Description of operation)
Next, the operation of the present embodiment will be described in detail with reference to FIGS.
FIG. 3 is a diagram showing an operation flowchart of the present embodiment. The user moves his / her mouth only to speak any character without speaking (step B1). During this period, the timer unit 2 notifies the image photographing unit 1 of a photographing instruction signal every time a fixed time interval (1 second) elapses. The image capturing unit 1 captures the user's mouth every time this notification is received (step B3), and transmits the captured image to the image analysis unit 4 (step B4). In addition, since the shooting interval of images is set to a 1-second interval longer than that of words such as normal conversation, the user needs to move his / her mouth so as to divide characters and speak according to the interval.

  FIG. 4 is a flowchart illustrating the operation of the image analysis unit 4. The image analysis unit 4 receives the image captured by the image capturing unit 1 (step C1), takes out one image data (image) from the basic mouth image database 5 (step C2), and compares the matching between the two to match. The degree of is numerically expressed (step C3). This is repeated for all points in the image in the basic mouth image database 5 (step C4), the voice corresponding to the mouth shape having a high degree of similarity is determined, and the character data (characters) is transmitted to the character string converter 6. (Step C5).

  FIG. 5 is a diagram showing an operation flowchart of the character string converter 6, and FIG. 6 is a diagram showing a data structure of the vocabulary database. The character string conversion unit 6 performs a process of collectively converting the received character data of one character and one character into character string data (character string). The character string conversion unit 6 combines the character information received from the image analysis unit 4 in the order received until the user presses the recognition start notification switch 9 to form a character string (step D1 and step D2). When the recognition start notification switch 9 is pressed, the character string conversion unit 6 reads one data from the vocabulary database 7 (step D3). As shown in FIG. 6, the data read at this time is composed of sound data composed of vowels, “n” and the like emitted from the mouth shape, and output word data paired with the sound data. The character string conversion unit 6 compares the sound data with the character string created from the characters received from the image analysis unit 4, digitizes the degree of matching (step D4), and holds it as a pair with the output word data. This is performed for all the data in the vocabulary database 7 (step D5). When the completion is completed, the values of the output word data are arranged in descending order, and the output word data having the highest coincidence is output to the speech synthesis output unit 10. Output to.

  The speech synthesis output unit 10 outputs the received output word data to the earphone 12. The user listens to the content, and if the result is intended by the user, the user presses the recognition OK button of the recognition result determination switch 13. The notification by the recognition OK button is notified to the speech synthesis output unit 10, and the output word data is output from the speech synthesis output unit 10 to the external output unit 11. If the result is not intended by the user, the user presses the recognition NG button of the recognition result determination switch 13. The notification by the recognition NG button is notified to the character string conversion unit 6, and the character string conversion unit 6 outputs the output word data with the next highest matching degree to the voice synthesis output unit 10 by this notification, and the voice synthesis output unit 10 again The output word data is output to the earphone 12, the determination by the user is performed, and the same processing as described above is performed. With the above processing, it is possible to correctly output the content to be spoken without the user uttering.

(Other embodiments)
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 7 is a diagram showing a second embodiment of the present invention. In the second embodiment, image recognition and speech synthesis are enabled by mouth movements close to speech in normal conversation, and an operation for notifying the timing of character conversion from the user is unnecessary.

  In the present embodiment, the recognition start notification switch 9 as shown in FIG. 1 is unnecessary, and an image analysis unit (hereinafter referred to as “continuous image analysis unit”) 14 that continuously analyzes images is provided. The timer unit 15 sends a signal to the image capturing unit 1 every n seconds with a time interval shorter than 1 second, and the image capturing unit 1 captures the shape of the user's mouth every n seconds, and the continuous image analyzing unit 14 Are sent continuously. The continuous image analysis unit 14 is configured to notify the character string conversion unit 6 of the start of conversion, and does not require the recognition start notification switch 9 as shown in FIG.

  For this reason, the continuous image analysis unit 14 adds the function of continuously comparing the image captured by the image capturing unit 1 with the image of the basic mouth image database 5 by capturing images at a time interval shorter than 1 second. When shooting with overlapping mouth shapes, unnecessary image information is thinned out to eliminate this overlapping subsequent image. For example, when three or more identical characters are recognized in succession, Recognize, thin out all subsequent characters, for example, all but the first character from the same consecutive characters, and output the first character data to the character string converter 6 and start recognition for the character string converter 6 It has a function to notify.

FIG. 8 is a diagram showing an operation flowchart of the second embodiment. The operation of the second embodiment will be described below with reference to FIGS.
Compared with the first embodiment, the second embodiment is that the shooting interval notified from the timer unit 2 is n (<1) seconds, and the images shot by the image shooting unit 1 are continuous. Other than the image analysis unit 14 inputting and recognizing the character data and notifying the character string conversion unit 6, and the continuous image analysis unit 14 notifying the character string conversion unit 6 of the start of recognition of the character string data etc. Is the same as in the first embodiment.

Referring to FIG. 8, in the second embodiment, a character string stored for internal processing of the continuous image analysis unit 14 is initialized (step E1), and then an image photographed by the image photographing unit 1 is The image is sent to the continuous image analysis unit 14 (step E2). Similar to the image analysis unit 4 of the first embodiment, the continuous image analysis unit 14 reads one image of the image of the basic mouth image database 5 and the received image (step E3), performs matching comparison, and performs matching. The degree is digitized (step E4). This is performed for all cases in the basic mouth image database 5 (step E5), and when all cases are checked, the character string corresponding to the one having the highest degree of matching is already stored. The character data is output to the character string conversion unit 6 while being coupled backward (step E6). Here, as long as three or more identical characters are not continuously combined, the processing of step E2 is repeated and the combination of the character strings for internal processing is continued (step E7, NO).
In step E7, when three or more identical characters are continuously combined (step E7, YES), it is recognized as a break of the user's utterance, and all but the first one character is thinned out from the same consecutive characters (step E8) The head character data is output to the character string converter 6 and the start of recognition is notified to the character string converter 6 (step E9).
The subsequent operation of the character conversion unit 6 performed based on the character data by the operation of the continuous image analysis unit 14 and the notification of the start of recognition is the same as the operation of the first embodiment.

  As described above, in the first embodiment, the user must move the mouth at intervals of one second, and thus the mouth movement corresponding to the speech cannot be smoothly performed. By making the n second value of the notification interval of the timer unit 2 smaller than that, the user can move the mouth without leaving a one-by-one interval, and also control the start of recognition. It is possible to output specific character data continuously by gestures of normal conversation such as stopping the speech or closing the mouth, etc., and there is no need for the user to notify by switch operation every predetermined character string, so smoother silence It is possible to make a call.

It is a figure which shows 1st Embodiment of the voiceless communication system of this invention. It is a figure which shows the image imaging means of 1st Embodiment. It is a figure which shows the operation | movement flowchart of 1st Embodiment. It is a figure which shows the operation | movement flowchart of the image analysis part. It is a figure which shows the operation | movement flowchart of the character string conversion part. It is a figure which shows the structure of the data of a vocabulary database. It is a figure which shows the 2nd Embodiment of this invention. It is a figure which shows the operation | movement flowchart of 2nd Embodiment. It is a figure which shows the prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image photographing part 2 Timer part 3 Illumination part 4 Image analysis part 5 Basic mouth shape image database 6 Character string conversion part 7 Vocabulary database 9 Recognition start notification switch 10 Speech synthesis output part 11 External output part 12 Earphone 13 Recognition result judgment switch 14 Continuous Image analysis unit

Claims (3)

An image photographing unit for photographing the mouth, an image analyzing unit for generating character data from the photographed image data, a character string converting unit for generating character string data from the character data, and synthesizing voice data from the character string data A speech synthesis output unit, a speech output unit that converts the speech data into speech, and a recognition result determination switch that selects whether or not to output the character string data to an external output unit, the image analysis unit The character string conversion unit outputs a character string conversion unit from the image analysis unit by thinning out the subsequent character data of the continuous character data as a character string delimiter when a certain number of the same character data continues. A voiceless communication system , wherein character data up to the input time is converted into character string data and output .   A basic mouth image database having a relationship between a shape of a mouth portion of image data and character data as a database, wherein the image analysis unit refers to the basic mouth shape image database and calculates character data from the shape of the mouth portion of the photographed image; The voiceless communication system according to claim 1, wherein the voiceless communication system is generated. 2. A mask for covering the mouth portion and photographing the mouth portion, wherein at least an illumination portion for illuminating the mouth portion and the image photographing portion for photographing the mouth portion are arranged inside the mask. Or the voiceless communication system of 2.

Images