WO2020236103A1

WO2020236103A1 - A system and a method for speech tracking for hearing impaired individuals

Info

Publication number: WO2020236103A1
Application number: PCT/TR2020/050227
Authority: WO
Inventors: Mete YAĞANOĞLU
Original assignee: Atatürk Üni̇versi̇tesi̇ Bi̇li̇msel Araştirma Projeleri̇ Bi̇ri̇mi̇
Priority date: 2019-05-23
Filing date: 2020-03-20
Publication date: 2020-11-26
Also published as: TR201907878A2

Abstract

The invention relates to a system which converts the speech to text and projects them onto any surface at which the user is directed to, or transmits them to the users fingertips as a vibration signal according to a defined vibrational alphabet, thereby allowing hearing impaired individuals to follow conversations in real time more effectively, and a method of operation for this system.

Description

A system and a method for speech tracking for hearing impaired individuals Technical Field

The invention relates to a system and a method that enables hearing impaired individuals to follow conversations.

The invention in particular relates to a system which allows the hearing impaired individual to follow conversations in real time by translating the conversations in the environment and projecting on any surface or transmitting as a vibration signal to the fingers, and a method for operation of this system.

State of the Art

Today, hearing aids can be used by individuals who are found to have hearing loss, so that individuals with hearing impairment can easily follow the conversations around them. However, the hearing aid is not effective in individuals who have lost hearing sensitivity to a large extent. The mentioned situation reveals the need for communication without the sense organs of these individuals.

In the current technique, hearing impaired individuals communicate with other individuals by using sign language consisting of gestures and mimics or by writing. However, the fact that sign language is a language that needs to be learned and written communication results in time loss and is not suitable for all environments have made an innovation necessary in the relevant field.

Patent application no. WO2018144896 has been encountered in a survey in the state of the art. The application relates to a device that can be attached to the wearer's clothing and can receive sounds from the environment and convert them to text and record them, then project them onto any smart device screen. The mentioned invention limits the use of the device by requiring an additional smart device screen to follow the converted text.

Another study found in the state of the art is patent application no. US20140337023A1 . The mentioned application refers to a system that determines the target speaker using image processing methods, then takes the speaker's audio signal and converts it into text, reflecting the said text onto the inner surface of glasses worn by the user. Tracking the text through the glasses by user creates difficulty in continuing conversations. In addition, the glasses in the system do not provide comfort due to obstructing the user's view.

As a result, due to the negativity described above and the inadequacy of the existing solutions on the subject, an improvement in the relevant technical area has been made necessary.

Object of the Invention

The invention is inspired by existing situations and aims to solve the disadvantages mentioned above.

The main aim of the invention is to enable hearing impaired individuals to follow conversations in the environment in real time, both in the field of education and in daily life.

Another aim of the invention is to ensure that the written text of the conversations can be projected on any surface in real time so that hearing impaired individuals can follow them, without the need for computers, cell phones, tablets or any other additional devices.

Another aim of the invention is to enable speech to be transmitted to the fingers of hearing impaired individuals using a defined vibrational alphabet, thus enabling hearing impaired individuals to follow speech more effectively.

Another aim of the invention is to enable hearing impaired individuals who miss real-time speech tracking to be able to see the speech text.

The structural and characteristic features and all advantages of the invention will be understood more clearly through the detailed explanation written with the following figures and references to these figures, and therefore assessment should be made by considering these figures and detailed explanation.

Brief Description of Drawings

Figure 1 is a block diagram representation of the speech tracking system of the invention for individuals with hearing impairments.

Figure 2 is a representation of the hand wearable device in the system of the invention.

Figure 3 is the flow chart of the signal processing steps in the speech tracking method of the invention for hearing impaired individuals. Description of Part References

11. Microphone

12. Processor

2. Forehead piece

21. Wireless module

22. Projection lens

3. Hand wearable device

31. Wireless module

32. Vibration motor

33. Screen

Detailed Description of the Invention

In this detailed description, the preferred structures of speech tracking system and method of the invention for hearing impaired individuals are explained only for a better understanding of the subject.

The invention, a representation of which is given in Figure 1 ; is a system that enables individuals with hearing impairments to follow conversations in real time; comprising

• fastened to the wearer's clothing,

o a microphone (1 1 ) that receives and transmits ambient sounds as an analog audio signal,

o a processor (12) that receives the analog audio signal transmitted by the microphone (1 1 ) and converts sounds to speech text and speech signal using the software installed in it and transmits them wirelessly

• a forehead piece (2) worn on the wearer's forehead, comprising

o a wireless module (21 ) that receives and transmits speech text transmitted from the processor (12),

o a projection lens (22) projecting the speech text received by the wireless module (21 ) to any surface to which the user is directed

• at least one wearable device (3) worn on the user's hand, comprising

o the wireless module (31 ) that receives and transmits the speech signal transmitted from the processor (12), o a vibration motor (32), which receives the speech signal transmitted from the wireless module (31 ) and converts it to a vibration signal in accordance with a specified vibration alphabet.

In an alternative embodiment of the invention, a screen (33) can be introduced on the wearable device (3), and the speech signal transmitted from the processor (12) can be projected onto the screen (33) as text. In addition, the history of the last conversation can be displayed on the screen (33). In this way, the user can capture the missed parts of the conversation by looking at the screen (33).

The working method of the invention is as follows:

• receiving the ambient audio signal by the microphone (1 1 ) and transmitting it to the processor (12),

• by the processor (12), sampling the analog audio signal using the software installed in it and converting it to digital audio signal, preprocessing the sampled audio signal to increase clarity, and extracting the cleaned audio signal from the important features for the audio signal through feature extraction,

• by the processor (12), the classification of the digital audio signal by comparing it with the installed language model and acoustic model, and the conversion of the classified audio signal into speech text after the feature selection process and the generation of the vibration signal suitable for speech text.

• receiving the speech text produced in the processor (12) by means of the wireless module (21 ) in the forehead piece (2) and transmitting it to the projection lens (22) in the forehead piece (2),

• projecting speech text by the projection lens (22) onto any surface towards which the user is directed to,

• receiving the vibration signal generated by the processor (12) through the wireless module (31 ) in the wearable device (3) and transmitting it to the vibration motor (32) in the wearable device (3),

• receiving the vibration signal transmitted from the wireless module (31 ) by the vibration motor (32) and transmitting it to the user's fingers by means of vibrations in accordance with a specified vibration alphabet.

The vibration alphabet used in the invention is composed of 22 characters by omitting Turkish characters ( , G, I, J, O, §, LI) in order to avoid complexity. Each character has a specific finger and amount of vibration. The head and index fingers are used for frequently used letters, while the other letters were distributed to the remaining fingers in accordance with their frequency of use. For example, the word "Merhaba" is coded as:

• M: Right hand thumb - 1 vibration,

• E: Left hand index finger - 2 vibrations,

· R: Right hand little finger - 2 vibrations,

• H: Left hand ring finger - 1 vibration,

• A: Left hand index finger - 1 vibration,

• B: Right hand little finger - 1 vibration,

• A: Left hand index finger - 1 vibration. Thereby, the user will be able to make sense of the vibration signals that is transmitted to his hand, thus enabling speech tracking in a more effective way.

Claims

1. A system that allows hearing impaired individuals to track speech in real time, characterized by comprising;

o a processor (12) that receives the analog audio signal transmitted by the microphone (1 1 ) and converts signal to speech text and speech signal using the software installed in it and transmitting them,

o a forehead piece (2) carrying a projection lens (22) projecting speech text transmitted from the processor (12) onto a surface towards which the user is directed.

2. A system according to claim 1 , characterized by comprising; the forehead piece (2) carrying a wireless module (21 ) which receives the speech text transmitted by the processor (12) and transmits it to the projection lens (22).

3. A system according to claim 1 , characterized by comprising; a wearable device (3) carrying a vibration motor (32) which receives the speech signal transmitted from the processor (12) and converts it to a vibration signal in accordance with a specified vibration alphabet.

4. A system according to claim 1 , characterized by comprising; the wearable device (3) carrying a wireless module (31 ) which receives the speech signal transmitted from the processor (12) and transmits it to the vibration motor (32).

5. A system according to claim 3 or 4, characterized by comprising; the wearable device (3) carrying a screen (33) which displays the speech text translated by the processor (12) to the user.

6. A system according to claim 1 , characterized by comprising; the projection lens (22) which has a moving head and is able to change orientation depending on the head movement of the user.

7. A method of operation for the system that allows hearing impaired individuals to track speech in real time, characterized by comprising; the process steps of

• receiving the ambient audio signal by the microphone (1 1 ) and transmitting it to the processor (12), • by the processor (12), sampling the analog audio signal using the software installed in it and converting it to digital audio signal, preprocessing the sampled audio signal to increase clarity, and extracting the cleaned audio signal from the important features for the audio signal through feature extraction,

• projecting speech text by the projection lens (22) onto any surface towards which the user is directed to.

8. A method according to Claim 7, characterized by comprising; the process steps of

9. A method according to claim 8, characterized by comprising; the process step of projecting the speech text received from the processor (12) by the screen (33) introduced on the wearable device (3).