KR20190069786A - A wearable device for the hearing impaired for deep learning-based sign language and voice translation - Google Patents

Abstract

The present invention discloses a wearable device for the hearing impaired for deep-learning-based sign language and voice translation. According to an embodiment of the present invention, a database unit can be formed, which recognizes sign language to be sensed through a camera and outputs a voice corresponding to sign language, recognizes a voice of a general person generated from the outside through a microphone and outputs a text corresponding to the voice, and collects sign language data necessary for sign language translation and performing repetitive learning on the collected data through a deep learning technique.

Description

TECHNICAL FIELD [0001] The present invention relates to a wearable device for a hearing impaired person for deep-learning-based sign language translation and voice translation,

The present invention relates to a wearable device for the hearing impaired for deaf-learning-based hydration and voice translation, and more particularly to a wearable device for the hearing impaired for deaf-learning-based hydration and voice translation, Recognizing the text corresponding to the speech and outputting the text corresponding to the speech to the device, and recognizing the sign language sensed through the camera, the text corresponding to the sign language is automatically completed through deep learning technology, one of artificial intelligence fields, And outputting the voice through the speaker of the speaker, respectively, so that the sign language-voice and the voice-text can be translated in real time.

According to the World Health Organization (WHO), 360 million people worldwide are deaf, and 32 million of them are children. In addition, the Ministry of Health and Welfare Disability Policy Division has published statistics on the disabled population. As a result, the number of people with hearing impairment in Korea is 269,000 as of 2015, and one in 200 people in our country suffers from hearing impairment. This is the next highest number of people with physical disabilities, which is equivalent to the population of Chuncheon City in Gangwon Province.

In general, hearing impairment refers to difficulties in listening to sounds due to interference with the hearing pathway from the ear to the understanding of the sound from the cerebrum. These hearing impairments result from inherited genetic or acquired accidents.

Hearing impaired people communicate in daily life through sign language, spoken language, and oral language. It visualizes the sound and communicates. The majority of the general public do not know about sign language. Therefore, it is difficult to expect communication between the deaf and the public through sign language. There is also a problem in the pace of communication. Finally, in the case of speech, it is an imperfect means of communication, because the hearing-impaired person understands what he or she says through the mouth shape of the general public, and ultimately the hearing-impaired person needs to understand what the public can understand when speaking to the public again. .

A key technology of the present invention is deep running. Deep learning is actually a kind of machine learning method for imparting intelligence to a machine. For example, it teaches a cat how to look at a machine by repeatedly repeating a specific example of 'various kinds of photographs' related to cats. Later, even if the machine shows pictures that have not been used for actual learning, the cat's appearance can be distinguished. It is important to understand that deep learning is big data.

A disadvantage of the translation function in addition to the artificial intelligence technology used in the present invention is that data of all cases can not be compared and analyzed. The present invention should be able to translate the sign language of all the wearers who wear the wearable device for the hearing impaired person for the deep learning-based hydration and voice translation. For this purpose, sign language data for learning should be collected.

The wearer should not feel any delay in using speed. This is because the degree of recognition of the discomfort increases as the delay time is felt. To do this, we process data using only consonants and numbers. If you do 'A' in consonants, you should extract the cases in the order you used most in relation to 'A' among the words or phrases you used. The technology used at this time is the RNN (Recurrent Neural Network) method among the deep learning techniques. The RNN scheme has high efficiency in automatic completion through recommendation of characters.

The present invention performs a voice-to-text translation function in addition to a hydration-voice translation function. This is to output the visual text that the hearing impaired can understand by inputting the voice of the general public.

People with hearing impairments have difficulty in breaking conversations with ordinary people. To solve these difficulties, there is a need for a wearable device for the hearing impaired person for the above-described deep learning-based sign language and voice translation, which improves the accuracy and speed of the translation function.

Korean Patent Publication No. 10-2016-0015723, "Bi-directional Sign Language Translation Method Performed by Bi-Directional Sign Language Translation Apparatus & Korean Patent Publication No. 10-2017-0084356, "Wearable Device for the Deaf" Korean Patent Laid-Open Publication No. 10-2014-0118003, "Lip motion-based sign language translation device with bidirectional display"

The present invention should help to improve the accuracy of the data prediction by translating the sign language used by the hearing impaired person without error. To this end, we provide wearable devices for the hearing impaired for deep-learning-based sign language and voice translation.

The present invention outputs the text of a deaf person's speech using the RNN technique, which has excellent performance in natural language processing, and translates the text into speech. This is about autocompletion through recommendation of characters, which provides faster sign language translation technology.

The present invention provides a wearable device for a hearing impaired person for deaf learning-based sign language and voice translation, which can be used to help a variety of hand movements from the restriction of the use of the hand in the viewpoint of the hearing impaired person,

The present invention provides a wearable device for a hearing impaired person for deaf learning based hydration and voice translation in which an optical part includes a prism for securing the visibility of the hearing impaired person.

The present invention provides a wearable device for a hearing impaired person for a deep learning-based sign language and voice translation that includes a touch-based LED switch as a component for determining sharpness of text output to a display.

The bidirectional sign language and speech translation unit according to an embodiment of the present invention includes a database unit that collects sign language data of a user and repeatedly learns the sign language data to increase accuracy of one-to-one correspondence between sign language and voice; A voice-text output unit for recognizing a voice of a general person generated from outside through a microphone and outputting a text corresponding to the voice; And a hydration-audio output unit for recognizing the hydration sensed through the camera and outputting a sound corresponding to the hydration.

The database unit may include progress from data collection to data training, which is the basis of the deep learning technique, and improvement of accuracy of translation according to repetition of training.

The sign language speech output unit may include a part for recognizing the sign language of the user (the hearing impaired person) photographed through the camera mounted on the apparatus and outputting text corresponding to the sign language to the prism and outputting the text as actual speech .

The voice-text output unit may include receiving a voice of a general person as a micro-input of the apparatus, converting the voice into one-to-one correspondence with the voice, and outputting the result to the prism.

A wearable device for a hearing impaired person for a deep learning-based hydration and voice translation according to an embodiment of the present invention includes a controller capable of controlling an operation of the wearable device; A power supply unit for supplying electric power for driving the wearable device; An image processing unit for processing an image photographed through a camera for sign language recognition in the bi-directional sign language and voice translation unit; And an optical unit designed so that there is no difficulty in daily life when the user wears the wearable device.

According to an embodiment of the present invention, there is provided an optical unit comprising: an LED switch unit for enabling adjustment of a screen of a display; A display unit for displaying a GUI screen serving as a user interface of the wearable device; And a convex lens and a prism portion for ensuring the convenience of the user.

In the convex lens and the prism portion, the wearable device has a small display. Therefore, in addition to a convex lens that enlarges the size of a screen that actually enters the eye, the wearable device refracts the light of the actual display, Lt; / RTI >

The GUI of the display unit includes a part for allowing the user to confirm the speech that the user desires to do in the form of text before voice output, and a part for converting the voice of the other party into text and outputting the text.

According to the embodiment of the present invention, the user of the wearable device for the hearing impaired person for the hydration and voice translation based on deep learning can accurately translate signatures in all cases and improve the prediction accuracy of data.

The wearable device for the hearing impaired person for deaf-learning-based hydration and speech translation according to the embodiment of the present invention is a device for deafening the hearing-impaired person displayed in text on the GUI of the display unit by using RNN technology And to provide a more rapid hydration-sound output technology through the automatic completion function using the recommendation of the characters.

According to the embodiment of the present invention, the wearable device for the hearing impaired person for the deaf learning-based hydration and voice translation has a hardware form of glasses, so that the hearing impaired person can restrict the use of the hand in the position of the hearing impaired person, And to provide a wearable device for helping the hearing impaired person to hydrate and translate the speech.

A wearable device for a hearing impaired person for deaf learning-based hydration and speech translation according to an embodiment of the present invention is a wearable device for a hearing impaired person, which includes an optical part having a prism for securing a visibility of a hearing impaired person, Device.

A wearable device for a hearing impaired person for deaf learning-based hydration and voice translation according to an embodiment of the present invention includes a touch-type LED switch for determining sharpness of text, a wearable device for helping the hearing- ≪ / RTI >

FIG. 1 is a block diagram illustrating an operation procedure of an LED switch unit 301 and a display unit 302 in an optical unit according to an embodiment of the present invention.
2 is a block diagram illustrating the operation of the convex lens & prism unit 400 in the optical unit according to the embodiment of the present invention.
3 is a block diagram illustrating the operation of the hydration-speech output unit 500 according to an embodiment of the present invention.
4 is a block diagram illustrating an operation of the voice-text output unit 600 according to an embodiment of the present invention.
5 is a block diagram illustrating an operation process of the database unit 700 according to an embodiment of the present invention.
FIG. 6 is a block diagram schematically illustrating the configuration of a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention.
FIG. 7 is an overall configuration diagram including a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention.
FIG. 8 is a block diagram showing a structure of a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention.
FIG. 9 is a block diagram showing a GUI to be output on a prism screen of a wearable device for a hearing impaired person for a deep learning-based hydration and voice translation according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and accompanying drawings. However, the present invention is not limited to the embodiments.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / Or additions.

As used herein, the terms "embodiment," "example," "side," "example," and the like should be construed as advantageous or advantageous over any other aspect or design It does not.

Also, the term 'or' implies an inclusive or 'inclusive' rather than an exclusive or 'exclusive'. That is, unless otherwise stated or clear from the context, 'x uses a or b.' Quot; refers to any of the natural inclusive permutations.

Also, the phrase "a" or "an ", as used in the specification and claims, unless the context clearly dictates otherwise, or to the singular form, .

Furthermore, the terms first, second, etc. used in the specification and claims may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The terminology used herein is a term used for appropriately expressing an embodiment of the present invention, which may vary depending on the user, the intent of the operator, or the practice of the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating the operation of the LED switch 301 and the display 302 in the optical unit according to the embodiment of the present invention.

Referring to FIG. 1, the LED switch unit 301 and the display unit 302 in the optical unit according to the embodiment of the present invention may be used in a wearable device for a hearing-impaired person for deaf- It provides the function to adjust the sharpness of the text to be displayed on the screen by using the LED switch.

To this end, the optical unit according to the embodiment of the present invention includes an LED switch 301 and a display 302.

FIG. 2 is a block diagram illustrating an operation process of the convex lens & prism 400 in the optical unit according to the embodiment of the present invention.

2, the contents output from the optical unit according to the embodiment of the present invention to the display unit 302 of FIG. 1 are transmitted through the convex lens & prism unit 400 to a depth- And provides a function of illuminating the screen of the wearer's eyes through the enlargement and refraction of the screen of the wearable device for the hearing impaired person.

To this end, the optical unit according to the embodiment of the present invention includes a convex lens & prism unit 400.

3 is a block diagram illustrating the operation of the hydration-speech output unit 500 according to an embodiment of the present invention.

Referring to FIG. 3, a sign language voice output unit 500 according to an embodiment of the present invention provides a function of inputting a sign language of a person with a hearing impairment and outputting it as a voice understood by the general public.

To this end, the deep learning-based hydration-sound output unit using the sign language data according to the embodiment of the present invention includes a sign language data unit 501, a sign language recognition unit 502, a text conversion unit 503, a text output unit 504, And an audio output unit 505.

The sign language data unit 501 indicates sign language information to be recognized as an input value in the sign language speech output unit.

The sign language recognition unit 502 provides a function of an interface for receiving an input value from the sign language speech output unit.

The text conversion unit 503 provides a function for converting the text corresponding to the sign language in one-to-one correspondence in the sign language-speech output unit.

The text output unit 504 provides a function of determining a text to be output as a voice using the RNN scheme based on the deep learning in the hydration-speech output unit.

The speech output unit 505 provides a function of outputting the result value output from the hydration-speech output unit to the prism as a voice that can be understood by the general public.

4 is a block diagram illustrating an operation of the voice-text output unit 600 according to an embodiment of the present invention.

Referring to FIG. 4, the voice-text output unit 600 according to the embodiment of the present invention receives a voice 601 of a general person as an input value, and outputs a value in text corresponding to the voice do.

To this end, the voice-text output unit according to the embodiment of the present invention includes voice data 601, voice input unit 602, text conversion unit 603, and text output unit 604.

The voice data 601 represents voice information to be recognized as an input value in the voice-text output unit.

The voice input unit 602 provides a function of an interface for receiving input values from the voice-text output unit.

The text conversion unit 603 provides a function for converting the text corresponding to the voice contents one-to-one in the voice-text output unit.

The text output unit 504 performs a function of providing voice contents that are converted to text in the voice-text output unit and output to the screen.

5 is a block diagram illustrating an operation process of the database unit 700 according to an embodiment of the present invention.

Referring to FIG. 5, data collection and training is performed to utilize the deep learning technique according to an embodiment of the present invention.

For this purpose, a relationship diagram 703 of the accuracy with respect to the number of times of repeated training, together with the hydration data part 701 and the data training part 702 necessary for using the deep learning technique according to the embodiment of the present invention, .

The data collection unit 701 provides a function of collecting consonants and numerical data necessary for deep running in the database unit 700.

The data training 702 provides a training function for performing a data comparison function to be used in the deep learning technique in the database unit 700.

The relationship between the number of training times and the degree of accuracy (703) indicates that the accuracy value is increased according to the number of times of training of the collected data and the progressed time.

FIG. 6 is a block diagram schematically illustrating the configuration of a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention.

Referring to FIG. 6, the operation of a wearable device for a hearing impaired person is determined for deep learning-based hydration and voice translation according to an embodiment of the present invention.

To this end, a wearable device for facilitating hydration and speech translation for a hearing-impaired person based on a deep learning according to an embodiment of the present invention includes a power source unit 72, an image processing unit 74, an optical unit 80, And a translation unit 90.

The power supply unit 72 provides a power supply function of the wearable device for helping the deaf learning-based deaf and hard of hearing person to do hydration and voice translation.

The image processing unit 74 provides a function of determining contents to be output on the screen of the wearable device that assists the deaf-mute-based deaf-mute person's hydration and voice translation.

The optical unit 80 provides a function of displaying an output value on the eye of the user in the form of a glasses to be output through a wearable device for helping deaf and hard of hearing handicapped and voice translation.

The bi-directional sign language and speech translation unit 90 represents an application and a method of a deep learning technique used for translation in a wearable device for the hearing impaired for the deep learning-based sign language and voice translation.

FIG. 7 is an overall configuration diagram including a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention.

7, a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention uses an input / output interface such as a microphone, a speaker, and a camera, a bidirectional hydration and voice translator 30, Use deep-learning technology.

To this end, the input / output interface according to the embodiment of the present invention includes a voice input / output unit 22 and a sign language recognition unit 24.

The voice input / output unit 22 receives the voice of a general person as an input value through a microphone, and translates the voice of a deaf person through the speaker.

The sign language recognition unit 24 takes the sign language data of the hearing impaired person through the camera and recognizes the sign language data as new data.

FIG. 8 is a block diagram showing a structure of a wearable device for a hearing impaired person for deep-learning-based hydration and voice translation according to an embodiment of the present invention.

Referring to FIG. 8, each component to be actually placed in the form of glasses of a wearable device for the hearing impaired for deaf learning-based hydration and voice translation according to an embodiment of the present invention is shown.

Each component according to the embodiment of the present invention includes a power source unit 101, a control unit 102, an image processing unit 103, an optical unit 204, and a camera 109 And a voice input / output unit to be located on the left side of the wearable device.

FIG. 9 is a block diagram showing a GUI to be output on a prism screen of a wearable device for a hearing impaired person for a deep learning-based hydration and voice translation according to an embodiment of the present invention.

9, a GUI screen to be output according to an embodiment of the present invention is displayed on a display screen of a wearable device for a hearing impaired person for the deep learning-based sign language and voice translation, The user provides a function of displaying a screen showing the words that the user wants to say through sign language, and a screen showing the entire contents to be automatically completed through the case that the user has mainly said through some of the words that the user wants to say.

To this end, the GUI screen of the wearable device for the hearing impaired user for the deep learning-based hydration and voice translation according to the embodiment of the present invention includes a general voice output unit 51, a user hydration input unit 52, an RNN scheme selection unit 53, an autocomplete case unit 60, and a current device state unit 54.

The general voice output unit 51 outputs a general voice input value received via a microphone and performs a function of allowing the user to view the voice input value.

The user sign language input unit 52 converts the content of sign language recognized through the camera into text and displays the text on a GUI screen so that the user can view the sign language.

The RNN scheme selection unit 53 performs a function of displaying a completed text through an automatic completion function of a character even if the user does not say everything he wants to say through sign language.

The automatic completion case unit 60 performs a function of outputting words or words frequently used by the user on the GUI screen by using the RNN method among the deep learning techniques so that the user can select them.

The device status unit 54 performs a status display function showing whether the wearable device for a hearing impaired person is currently operating normally, for the dehydration-based hydration and voice translation through the GUI screen.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

30: bi-directional sign language and speech translation unit 31: sign language-voice output unit
32: voice-text output unit 33: database unit
20: Wearable device for the hearing impaired
22: voice input / output unit 24: sign language recognition unit
11: The process of inputting through the voice microphone of the general public and outputting the sign language of the hearing impaired person as voice
12: Process of outputting through speech text of general public and sign language input process of deaf person

Claims (10)

An optical unit which plays a role of a basic hardware necessary for a user to use a GUI (Graphic User Interface)
A wearable device for the hearing impaired for deep-learning-based hydration and speech translation. The method according to claim 1,
The optical unit includes:
An LED switch unit for adjusting a brightness of a display screen according to a user;
A display unit for displaying a GUI screen through LED light; And
A convex lens and a prism portion for enlarging and refracting the contents of the outputted display
A wearable device for the hearing impaired for deep-learning-based hydration and speech translation. A hydration-audio output unit for recognizing the hydration sensed by the camera and outputting a sound corresponding to the hydration;
A voice-text output unit for recognizing a voice of a general person generated from outside through a microphone and outputting a text corresponding to the voice; And
A database portion for collecting sign language data necessary for sign language translation and for repeating the collected data through a deep learning technique
Lt; RTI ID = 0.0 > a < / RTI > The method of claim 3,
The sign language-
A sign language recognition unit for recognizing sign language sensed through the camera;
A text conversion unit for converting the text into text corresponding to the recognized sign language; And
A voice output unit for outputting voice corresponding to sign language based on the generated text,
And a bi-directional sign language and speech translation unit. 5. The method of claim 4,
Wherein the text conversion unit comprises:
It converts the sign language of the user into text and then uses the RNN method of deep learning technology to automatically complete the words of the hearing impaired.
A wearable device for the hearing impaired for deep-learning-based hydration and speech translation. The method of claim 3,
The voice-text output unit includes:
A voice input unit for recognizing voice through the microphone;
A text conversion unit for generating and converting a text corresponding to the recognized voice; And
A text output unit for outputting text to a display
And a bi-directional sign language and speech translation unit. The method of claim 3,
The database unit,
A sign language data part for collecting sign language data of the user and other users;
A data training unit for repeatedly learning the collected data by a deep learning technique to increase the accuracy of new incoming data and text to be output,
And a bi-directional sign language and speech translation unit. 8. The method of claim 7,
The database unit,
A relationship in which the accuracy of sign language and one-to-one correspondence of text increases in proportion to the number of times of repeated learning of the data in the data training section
And a bi-directional sign language and speech translation unit. A power supply unit for supplying power to a wearable device for the hearing impaired for deep-learning-based hydration and voice translation; And
An image processing unit for processing sign language images recognized through a camera and image contents in a GUI interface of a display
A wearable device for the hearing impaired for deep-learning-based hydration and speech translation. 10. The method of claim 9,
In the GUI of the display,
A voice-text output unit for outputting words to be spoken by the other party;
A sign language voice output unit for allowing the user to say a word; And
Autocomplete case that suggests text based on user's most used words through some consonants that user selected
A wearable device for the hearing impaired for deaf learning-based sign language and voice translation.

Images