KR100307229B1 - Method for outputting the finger language by speech recognition for a deaf-mute - Google Patents

Abstract

The present invention relates to a speech recognition sign language output method for deaf. An interface step (S100) of driving information use, communication of cultural life, and use of a communication medium as options desired by a user for communication with a hearing-language-disabled user and the general public; When the general public communicates to the hearing and speech impaired person in language, the voice recognition input step into the speech recognition sign language output system for the deaf through a PC microphone, a built-in microphone, a small microphone (S200); A voice synthesis input / output step (S300) of converting text into digital audio by voice synthesis (TTS) to output sound through a speaker or to reproduce the converted audio, and to convey an intention of outputting the speech to the general public by a person with a disability; And animation adjustment data (ACD) system to output sign language images in language and to display texts in real time using speech synthesis. And the sign language and text output step S400 for executing the information medium. Therefore, the hearing and speech impaired can receive information media such as TV, radio, internet and PC communication, and cultural life such as movies and music, and communicate with the general public through the communication media of ordinary telephones or mobile phones and lightweight PCs. have.

Description

Method for outputting the finger language by speech recognition for a deaf-mute}

The present invention relates to a speech recognition sign language output method for deaf, and in particular, for the satisfactory communication with a general person who is deaf or speech impaired, a speech synthesis, speech recognition, and animation sign language output software program is used for hearing, The interface stage between speech-impaired persons and the general public, the stage of speech recognition input, the stage of text-to-speech (TTS) input and output, and the sign language and text output stages of animation are provided to TV, radio, internet and The present invention relates to a speech recognition sign language output method for deaf children to provide information media such as PC communication and cultural life such as movies and music, and to use communication media such as general telephones and mobile phones.

Currently, the sign language interpreter system, which is being developed by a domestic university research institute, runs on a high-capacity system and outputs sign language through specially manufactured gloves.

However, because it requires a high-capacity computer system and the specially crafted gloves, it cannot be operated on a light weight computer or a hand PC (HPC). The sign language interpretation system is currently capable of interpreting only 400 words and is compatible and versatile. There is a weak problem.

The present invention has been made to solve the above-mentioned problems of the prior art, the object of the present invention is to provide a smooth communication with the general public, the deaf or speech-impaired people, and to the deaf and speech-impaired people, TV, radio, the Internet and It provides information media such as PC communication and cultural life such as movies and music.The lightweight PC such as hand PC (HPC) enables the hearing and language disabled to communicate anywhere and provides information media and cultural life. It provides communication media such as general telephones and mobile phones to people with speech impairments, and provides speech recognition sign language output method for deaf to make 'Sullivan' for the hearing and speech impaired run on a multimedia operating system (OS).

1 is a flow chart showing a speech recognition sign language output method for deaf according to the present invention.

2 is a flow chart showing the interface steps of FIG.

3 is a flowchart illustrating a voice recognition input step of FIG. 1.

4 is a flow chart showing the voice synthesis (TTS) input and output steps of FIG.

5 is a flowchart showing the animation sign language and the text output step of FIG. 1.

6 is a flowchart showing the interface with the hardware that Sullivan (Sullivan) implementing the speech recognition sign language output method for the deaf run on a general PC, mobile computer and HPC.

TTS: Text-To-Speech

HPC: Hand PC

In order to achieve the above object, the present invention is based on the conditions and environment of the hearing and speech impaired user communication with the general public, non-disabled people, information media such as TV, radio, PC communication and the Internet and culture of movies, music, etc. An interface step (S100) of providing a life and driving a basic interface with communication options such as a general telephone and a mobile phone by setting options desired by the user; When the general person communicates the speech to the user, such as the deaf or speech impaired person, the voice recognition input step is input to the speech recognition sign output system (Sullivan) for the deaf through a PC microphone, a built-in microphone, and a small microphone (S200). ); Text-To-Speech (TTS) can be used to convert plain text into digital audio and then store the sound through the computer's speakers or to store the converted audio for later playback. A voice synthesis (TTS) input / output step (S300) for performing a function of delivering an output intention to the general person who is the non-disabled person by voice; And a system called Animation Control Data (ACD) to define a real-time rendering system for outputting a sign language image in language and quickly displaying the animation control data, wherein the text is the TTS technology. Implements a small program that supports text in real time, and the language transmitted through the voice recognition input step is a doctor or the information medium, etc. Provides a speech recognition sign language output method for deaf, characterized in that consisting of a sign language animation text output step (S400).

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. 1 is a flow chart showing a speech recognition sign language output method for deaf according to the present invention. The speech recognition sign language output method for the deaf uses a speech synthesis, speech recognition, and animation sign language output software program for hearing and speech impaired people with non-disabled people. An interface stage, a speech recognition input stage, a text-to-speech (TTS) input / output stage, and an animation sign language / text output stage are included.

2, the interface step is

Reads the environment file (step S1), waits for a sign language output animation (step S2), waits for a voice recognition command (step S3) and starts a multithread (step S4); The multi-thread is started (step S4) while being fed back to the output animation waiting state and starting the user command interface (step S5). After starting the user command interface, check 'Would you like to talk?' (Step S6) and if desired, start the voice recognition input / output system (step S7) and search the language search system (step S8) to output the animation sign language. After one (step S9), the user text (Text) is input (step S10) to output a voice (step S11) and the multithread is terminated (step S16). After starting the user command interface, check 'Do you want to use the information medium?' (Step S12) and when the information medium is used, the voice recognition input / output system is driven (step S13), and the language search system is searched. (Step S14) After outputting the sign language of animation (step S15), the multithread ends (step S16). After ending the multithread, the system waits (step S17).

Referring to Figure 3, the voice recognition input step

The interface between the user, such as the deaf or speech impaired person, and the general person who is the non-disabled person is initialized (step S20), a voice data package is waited (step S21), and a voice data searcher is waited (step S22). After initializing the interface, voice data is initialized (step S23), the multithread is initialized (step S24), and voice recognition is performed using a PC microphone and a voice recognition standby device which is a built-in microphone or a small microphone (step S25). (Step S26). The voice data package is searched after the voice recognition input, the voice data package wait and the voice data searcher wait (step S22) (step S27). After searching for the voice data package (step S27), the animation sign language text is output (step S28) and the voice data package is waited (step S29) while waiting for a voice recognition input for continuous execution (step S21).

Referring to FIG. 4, the voice synthesis (TTS) input / output step

Initialize the interface between the user, such as the hearing and speech impaired person, and the general person who is the non-disabled person (step S30), wait for a voice data package (step S31), and initialize voice data (step S32) and the multithread. After performing initialization (step S33) and waiting for a voice synthesis (TTS) output of a keyboard or the like (step S34), the voice synthesis (TTS) is output through the user's keyboard input (step S35). After the voice synthesis (TTS) output (step S35) and waiting for the voice data package (step S31), a voice searcher is driven (step S36) to retrieve a voice data package (step S37) and voice is transmitted to the PC speaker and other internal speakers. Output (step S38). After the speaker voice output, the voice data package is waited (step S31) to perform continuous voice synthesis (TTS) input and output.

Referring to Fig. 5, the animation sign language / text output step is

Read the environment file and wait for command execution (step S40), and control animation control data (ACD) (step S41) to wait for an animation sign language character (step S42) and the animation control data (ACD) The user option is awaited under control of step S43. After waiting for the user option, the character size change option is performed (step S44), and the text output change option is performed (step S45), and the check box 'Do you want text output?' (Step S46) waits for the text output ( In step S47), a character change option is performed so that the user can change to a desired character (step S48). After performing the text output wait step, the character size change option (step S44) and the character change option S48, a voice recognition is input by an event (step S49) to drive a searcher (step S50). After the searcher is driven, the animation data search is driven (step S51) to output an animation sign language (step S52), the text output data search is driven (step S53) and text is output (step S54). After the animation sign language output (step S52) and the text output (step S54), the apparatus waits for the animation sign language / text output wait or voice synthesis (TTS) step for continuous execution.

FIG. 6 is a flowchart illustrating a hardware interface between a Sullivan, which implements a speech recognition sign language output method for deaf, and a PC, a mobile computer, and an HPC.

The language used for producing Sullivan was written in 100% pure Java language, and the tools used for the development of Sullivan were IBM's Jike Compiler 1.2 and Sun Microsystems' Java Development Kit (JDK) 1.2. The Java package uses Java AWT, Java Swing, Java Database, Java Beans, CORBA, and Java Network. The voice editor uses Syntrillium's Coolidit.

The Sullivan hardware is a new platform optimized to run Shirvan on a variety of computing and consumer platforms, and provides a runtime tailored to precede Shirvan on a hardware platform without a host operating system. As a result, Shirley-Ban has been designed to be portable with several hardware devices with highly interactive and dynamic stability. Devices controlled and controlled by the Shirley-van hardware may drive a CPU, a memory, a liquid crystal (LCD) monitor, a keyboard, a microphone, a speaker, and other mobile computer input / output devices. Starting the Sullivan system implementing the speech recognition sign language output method for the deaf (step S70), driving the kernel device driver interrupt control for each system (step S71), and through the graphics card driver mounted in the system In order to display the image on the monitor, the Java window Toolkit Shirley-Ban client user interface (GUI) is driven (step S72) and the graphics card input / output of the mobile computer, the general PC, and the liquid crystal monitor is waited and displayed on the monitor screen. Output (step S73).

Waiting for Java awt event package keyboard input / output value (S74) Waiting for keyboard input / output value (step S75) Waiting for Java awt event Audio package sound input / output value (step S76) Waiting for microphone input (step S77), Waiting for sound card output and The speaker output wait is executed (step S78). The Java awt event package keyboard input / output value means a keyboard thread event and a keyboard event capable of input / output mounted in the system through awt event which is a Java basic package.

Java awt event After waiting for audio package sound input / output values. The Java Network package client kernel driver waits for a modem input / output value (step S79) and executes by waiting for input / output on a modem telephone line (step S80). The Java awt event Audio package sound input / output value is a Java basic package that can control the sound card through the sound card drive installed in the system. The method is a thread, stream, exception handling, signaling, etc. It supports mic input and speaker output. The Java Network Package Client Kernel Driver Modem I / O values are used to create networking applications or communication-related applications through the package Java.net package supported by Java. That is, a communication device such as a modem supports dialing, voice reception, and voice transmission signals of a communication medium such as a general telephone or a mobile phone.

The speech recognition sign language output method for the deaf according to the present invention was used as a development language using a standalone system with its own interpreter using the Java language, and the tools used were JDK1.1 and JDK1.2 from Sun Microsystems. Java Component Object Model, which runs on many platforms such as Microsoft's Windows, Macintosh, OS / 2, Unix, Solaris, and Netware. We introduced a new Java Speech Object Model using Java Com technology to implement speech recognition technology and speech synthesis (TTS) animation.

Sullivan, which implements the speech recognition sign language output method for the deaf, was developed as pure Java embedded with the Java Component Object Model and uses the following method.

Java Source (* .Java)-Java Compiler-Java Bytecodes (* .class)

Class loader bytecode Werifier-Java class Libraries-just-in-time compiler

Runtime System-Operating System-Hardware

With the above compile time and run time environment, Sullivan can be embedded in each operating system, and the Sullivan has its own Java Speech Object Model System. Therefore, it has the strength of having enough data capacity even in small capacity.

For reference, the Sullivan may have its own system like a hearing aid, and the Sullivan may be used by the hearing and language impaired as a lightweight system such as a small system hand PC (HPC).

As described above, since Sullivan, which implements the speech recognition sign language output method for deaf children according to the present invention, is made of Java language, it can be operated in multiple operating systems, and it is possible for the hearing and speech impaired person to It can communicate smoothly with the general public with disabilities, and can receive information media such as TV, radio, internet and PC communication, and cultural life such as movies and music. Can communicate through communication media, and the hearing and speech impaired person can use communication, information media, cultural life and communication media smoothly by using a lightweight PC such as a hand PC (HPC). .

Although the above has been described with reference to a preferred embodiment of the present invention, those skilled in the art will be able to variously modify and change the present invention without departing from the spirit and scope of the invention described in the claims below. It will be appreciated.

Claims (6)

Based on the condition and environment of the hearing and speech impaired user, it provides communication media with non-disabled people, information media such as TV, radio, PC communication and internet, and cultural life such as movies and music. An interface step (S100) of driving a basic interface with a communication medium setting options desired by the user; When the general person communicates the speech to the user, such as the deaf or speech impaired person, the voice recognition input step is input to the speech recognition sign output system (Sullivan) for the deaf through a PC microphone, a built-in microphone, and a small microphone (S200). ); Text-To-Speech (TTS) can be used to convert plain text into digital audio and then store the sound through the computer's speakers or to store the converted audio for later playback. A voice synthesis (TTS) input / output step (S300) for performing a function of delivering an output intention to the general person who is the non-disabled person by voice; And A system called Animation Control Data (ACD) defines a realtime rendering system for outputting a sign language image in a language and quickly displaying the animation control data, and the text is the TTS technique. Implement a small program that supports text in real time using the language, and the doctor or the information from the user, such as the hearing and speech impaired, through the sign language and output of the text transmitted through the speech recognition input step A sign language output method for deaf children, characterized in that comprises a sign language and text output step (S400) for performing a medium or the like. The method of claim 1, The interface step Reads the environment file (S1), waits for command execution (S2), waits for a sign language output animation (S2), starts a multithread (S3) while waiting for a voice recognition command (S4), and waits for the sign language output animation. A multi-threaded step (S4) while starting a user command interface (S5); After starting the user command interface, check the 'Would you like to talk?' (S6), if you want to have a conversation, run the voice recognition input / output system (S7), search the language search system (S8) and output the animation sign language (S9). After inputting the user text (S10) to output a voice (S11) and terminating the multi-thread (S16); After starting the user command interface, check 'Do you want to use the information medium?' (S12) and when the information medium is used, the voice recognition input / output system is driven (S13), and the language search system is searched (S14). (S16) terminating the multithread after outputting an animation sign language (S15); And After the multi-threaded, the speech recognition sign language output method for deaf, characterized in that the step of waiting for the system (S17). The method of claim 1, The voice recognition input step Initializing the interface between the user, such as the deaf or speech impaired person, and the general person who is the non-disabled person (S20), waiting for a voice data package (S21), and waiting for a voice data searcher (S22); After the interface is initialized, voice data is initialized (S23), the multithread is initialized (S24), and a voice recognition input is performed using a voice recognition standby device (S25), which is a PC microphone and a built-in microphone or a small microphone. (S26) step; Searching (S27) a voice data package after the voice recognition input, the voice data package wait, and the voice data searcher wait (S22); And Outputting the animated sign language text after searching the voice data package (S27) and waiting for a voice recognition input for continuous execution (S29), and waiting for the voice data package (S21). Speech recognition sign language output method for deaf. The method of claim 1, The voice synthesis (TTS) input and output steps Initialize the interface between the user, such as the deaf or speech impaired, and the general person who is the non-disabled person (S30), wait for a voice data package (S31), initialize voice data (S32), and initialize the multithread ( Performing a step S33) and waiting for a voice synthesis (TTS) output of a keyboard or the like (S34) and outputting the voice synthesis (TTS) through the user's keyboard input (S35); After the voice synthesis (TTS) output (S35) and the voice data package standby (S31), a voice searcher is driven (S36) to retrieve a voice data package (S37) and output voice to a PC speaker and other internal speakers (S38). ) step; And And a step (S31) of waiting for the voice data package to perform continuous voice synthesis (TTS) input / output after the speaker voice output. The method of claim 1, The animation sign language and text output step Read the environment file and wait for command execution (S40), control animation control data (ACD) (S41) to wait for an animation sign language character (S42), and control the animation control data (ACD). Waiting for a user option (S43); After waiting for the user option, a character size change option is performed (S44), and a text output change option is performed (S45), 'Want to print text?' (S46) and text output is waited (S47), Performing a character change option so that the user can change to a desired character (S48); Driving the searcher by inputting voice recognition (S49) after performing the text output waiting step, the character size changing option (S44) and the character changing option (S48); After driving the searcher, driving an animation data search (S51) to output an animation sign language (S52), driving a text output data search (S53), and outputting a text (S54); And After the animation sign language output (S52) and the text output (S54), the speech recognition sign language output for deaf, characterized in that the step of waiting for the animation sign language, text output standby or speech synthesis (TTS) step for continuous execution. Way. The method of claim 1, A computer-readable recording medium having recorded thereon a program for executing the interface step, the voice recognition input step, the voice synthesis (TTS) input / output step, and the animation sign language / text step.