JP2003522978A - Method and apparatus for converting sign language into speech - Google Patents

Abstract

(57) [Summary] A portable device converts an input based on a gesture from a signer into an audible sound in real time. The apparatus employs a portable main processor, for example, one of the portable computers commonly used today. For its inputs, the device uses data gloves and for its outputs it uses speakers. Dynamic and static gestures are classified by a Continuous Hidden Markov Model (CHMM) that allows robust and fast real-time classification of both static and dynamic gestures. A natural language processor is used to convert the gesture class into a grammatically correct phrase sequence. The speech synthesizer converts the phrase string into audible speech.

Description

Detailed Description of the Invention

The present invention relates to a signal language translator, and more particularly to a translator for directly converting sign language into colloquial language using a portable computer.

A data globe is used to classify sign language. In one prior art,
Static sign language is translated into letters or phrases and gestures are ignored.
Discrete Hidden Markov Model with Data Globe Input (Discrete H
idden Markov model) enables interactive learning and has been successfully used to train a series of gestures. This technology is called "On-lin
e, interactive learning of gestures
for human / robot interfaces ", Christop
her Lee and Yanghsheng Xu, IEEE International Conference on Robotics and Automation (1996), vol. 4, 2982-298
It is disclosed on page 7.

[0003] Nutra networks specifically trained by users have been shown to be capable of recognizing the small groups of letters presented by dynamic sign language. This technology is
multi-stage approach to fingersspelli
ng and gesture recognition ", R.G. Erensh
teyn and P.T. Laskov, Proc. Workshop on the
Integration of Gesture in Language
and Speech (Wilmington, DE, 1996).

Another prior art system uses colored gloves and camera-based image processing techniques to continuously track gestures. This system does not allow sign language and interferes with the user's movements with the video input system, the need to wear special colored gloves, and the need to stay in view of one or more cameras. This technology is called "Visual recognition of American Sig".
n Language using Hidden Markov model
s ", Thad Stringer, Master's thesis, The
Media Laboratory, MIT, 1995. Data gloves have been proposed to map hand gestures to text using neural networks. This technology is called "Glove
-Talk II: Mapping hand gesture to spe
ech using neural networks-an appro
ach to building adaptive interfaces "
, Sidney Fels, PhD thesis, University of Toronto, 1994. A huge amount of processing power is required for real-time processing using a neural network. Currently, a system that provides ergonomic compatibility for mobile use, real-time conversion of continuous gestures to voice, and convenient operation that allows a signer to communicate with a non-signer,
It does not exist in the prior art.

The mobile device converts a gesture-based input from a signer into audible voice in real time. The device employs a portable main processor, such as one of the portable computers commonly used today. For its input, the device uses a data glove and for its output a speaker. Dynamic and static gestures are continuous hidden Markov models (Continuous H model) that enable robust and rapid real-time classification of both static and dynamic gestures.
It is classified by the idden Markov Model (CHMM). A natural language processor is used to convert the gesture classes into grammatically correct phrase sequences. The voice synthesizer converts the word sequence into audible voice.

The present invention benefits both in portability and usability by using HMMs to classify gestures. Such models are generous and do not require much computational power. Thus, they handle changes in input format,
Proper classification can be generated. In addition, they are significantly less utilized in computing resources than, for example, neural networks. Using a data glove at the input and a speaker at the output provides a high degree of portability of the device.

Moreover, the use of data grove allows ports of relatively small bandwidth to be used. The same applies to the output of the voice engine. The voice engine receives textual or other symbolic output coming in through a port and can be synthesized by an inexpensive external processor system. Alternatively, the processing unit may already have a sound card with speech synthesis capabilities, such as many personal digital assistants (PDAs). In this way, the system can be formed as a retrofit of existing and future PDS units equipped with appropriate software.

The present invention will now be described with reference to the following schematic drawings, together with certain preferred embodiments,
As explained, it will be more fully understood. With reference to the drawings, the depicted items are exemplary and are intended only to illustrate the preferred embodiments of the invention and are the most informative and quick understanding of the principles and conceptual aspects of the invention. Emphasize that it is presented to provide what is believed to be the explanation given. In this regard, it is necessary to provide a basic understanding of the invention, and more detailed than the description taken with the drawings that makes apparent to those skilled in the art how some forms of the invention may actually be implemented. No attempt has been made to provide structural details of the invention.

Please refer to FIG. The data glove 130 and position sensor 110 apply hand position and placement data to the gesture recognition processor 120. The gesture recognition processor 120 classifies hand gestures into discrete symbols that can be identified by a phrase, and generates an output indicating the classified phrase in real time. This information may also be output where the confidence index generated by the classification is low. The classification information is then applied to the natural language processor 140. The natural language processor 140 converts the phrase into sentences and phrases that are completely in grammar. The sentences and phrases can be output in text or other more compact symbolic formats. The output of natural language processor 140 is applied to speech synthesizer 150. This speech synthesizer 1
50 generates an audio signal that can be output to the speaker 195. Alternatively,
This audio signal may be generated, for example, at port 160, which may be connected to headphones (not shown) that allow for personal use or use in noisy environments.
This may be especially beneficial if the signer is good at lip reading. Because the conversation can be completely secret to the non-lipreader.

The data glove 130 and position sensor 110 may be any electromechanical device capable of producing a signal in response to a sign language gesture. For example, an inertial sensor with a direct integrated signal may provide velocity and position information for various parts of the hand, such as the wrist or some or all fingers. Alternatively, data gloves currently on the market and used for control applications may be used. The types of inputs needed to form the actual device for this application will become more apparent with continued research in the field. Currently, the various prototypes mentioned above are
And velocity information can be extracted into a manageable data space (a reasonable number of independent inputs), which are applied to various types of recognition processors to classify sign language type gestures, Has been proven. Note, of course, a combination of inertial and encoder-based techniques may also be used.

The gesture recognition processor 120 may be based on a variety of different techniques that may classify the gesture input. The current technology in software and handsets is the Continuous Hidden Markov Model (CHMM
) Is the preferred approach. Another advantage of the CHMM classification technique is the fact that it tends to tolerate changes in input and relative values. Note that as the processor speed, integrated scale, size, and cost of computing hardware evolve, other classification techniques such as classification based on neural networks may also be appropriate.

The gesture recognition processor 120 outputs a class indicator for each recognized gesture. A stream of such indicators is
Applied to natural language processors that add missing words to form grammatical sentences and phrases. Since sign language does not necessarily include all the elements of normal speech, that is, obvious and basic grammatical components such as subjects and articles can be omitted, the natural language processor sends them to the speech synthesizer 150. Can be inserted before application. Natural language processor 140 identifies non-grammatical usages and corrects them. Such techniques are well developed for word processors and can be directly applied in urgent cases. Note that the natural language processor 140 is not essential, as non-grammatical speech corresponding to sign language may still be recognizable. Therefore, in training a natural language processor, it would be best not to make any modifications if the confidence in responding to changes is low. That is, the natural language processor may be adjusted to change only if the confidence measure for the considered change is high. This is because intelligible speech can be derived directly from the output of the gesture recognition processor.

Speech synthesizer 150 may be any phrase-to-audio converter such as a text-to-speech converter. The voice is preferably output to a small speaker or other audio converter. Note that the text does not have to be an intermediate organism in the rapid mode. However, it may facilitate the use of off-the-shelf devices such as existing text-to-speech converters.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above specific embodiments, and that the present invention can be embodied in other specific forms without departing from its spirit or basic characteristics. Will. Therefore, the present embodiments should be regarded as illustrative and non-limiting in all respects, and the scope of the present invention is shown not by the above description but by the appended claims, and thus the equivalent meaning and scope of the claims. All changes that come within are intended to be included therein.

[Brief description of drawings]

[Figure 1]   It is a figure of the portable sign language voice converter concerning one embodiment of the present invention.

─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5D045 AB30                 5L096 CA27 DA05

Claims (10)

[Claims] 1. A portable device for converting sign language to audible speech, the first data representing at least one human hand gesture indicating the gesture.
An electromechanical transducer translatable into a stream, a gesture classifier, and a speech synthesizer, the classifier being connected to receive the first data stream, the first data stream A second data stream indicating a first word corresponding to the gesture recognized as a sign language gesture by the gesture classifier may be generated according to the stream, and an audio signal corresponding to the word may be the voice signal. As produced by the synthesizer,
Connected to apply the second data stream to the speech synthesizer,
A device characterized by the above. 2. The apparatus of claim 1, wherein the electromechanical transducer is at least partially wearable on the human hand. 3. The apparatus according to claim 2, wherein the electromechanical transducer comprises a data globe. 4. A device according to claim 1, further comprising a sound emitting transducer connected to receive the audio signal so that the words are converted into audible speech. A device having. 5. The apparatus according to any one of claims 1 to 4, further comprising a natural language processor connected between the gesture classifier and the phonetic product, the natural language processor , A programmable processor programmed to insert at least one second word between a set of the first words to repair a non-grammatical structure of the sequence of the first words. A device characterized by the above. 6. Device according to claim 5, characterized in that it is formed as a modular unit, for example a unit wearable by the user. 7. The apparatus of claim 1, wherein the natural language processor is formed as a modular unit and is connected between the gesture classifier and the speech synthesizer, and the word is converted to audible speech. A sound emitting transducer connected to receive the audio signal, the natural language processor inserting at least one second sequence structure of the first words. A programmable processor, the electromechanical transducer being at least partially wearable on the human hand, and the sound emitting transducer being normal speech from a wearer of the modular unit. A device that is sufficiently powered to be heard by a person standing in the distance. 8. The apparatus of claim 1, wherein the natural language processor is formed as a modular unit and is connected between the gesture classifier and the speech synthesizer, and the audio signal is received. A port connectable to a sounding transducer so that the word is converted to audible speech, the natural language processor for repairing a non-grammatical structure of the first word sequence. A programmable processor programmed to insert at least one second word between a set of the first words, the electromechanical transducer being at least partially in the human hand. A device that can be mounted. 9. A method for converting sign language into audible speech, the method comprising: generating a first signal corresponding to a gesture of a human hand; classifying the first signal as a gesture and indicating a class. A second signal, a step of indicating a corresponding word according to the class, and a step of outputting the word in a human-perceptible format, wherein the classifying step is Continuous Hidden A method comprising applying a Markov model. 10. The method of claim 9, wherein the outputting step comprises applying the second signal to a speech synthesizer.