WO2020117089A1

WO2020117089A1 - Communication system for persons with impairments of hearing and vision

Info

Publication number: WO2020117089A1
Application number: PCT/RU2018/000794
Authority: WO
Inventors: Денис Сергеевич КУЛЕШОВ; Андрей Павлович БЫТЕЙЩИКОВ; Александр Валерьевич ГОРЮНОВ; Марк Викторович ЗОЛОТАРЁВ; Николай Николаевич КИРИЧЕНКО; Евгений Константинович ПОПОВ; Александр Николаевич СЕМЕНЕНКО; Александр Викторович Попов
Original assignee: Общество с ограниченной ответственностью "Нейроботикс"
Priority date: 2018-12-07
Filing date: 2018-12-07
Publication date: 2020-06-11

Abstract

The invention relates to dedicated medical devices for individual use which are intended for the rehabilitation of disabled persons, specifically to a communication system for persons with impairments of the sensory functions of hearing and vision (deafblind), namely a neurocommunicator, and can be used as a channel for deafblind persons to communicate with the outside world. The invention is aimed at expanding the capability of deafblind users in controlling their environment and makes it possible to increase the independence and self-sufficiency of the relevant cohort of persons with health-related disabilities by raising the functionality of the device. The system is a modular unit-based structure which is wirelessly connected. The system includes three units: a control unit, a monitoring unit and an information processing and output unit. The control unit is formed of a microcomputer with a computer platform using cross-modal recoding of information. The monitoring unit is in the form of a functional testing module for the deafblind which provides functional, psychological and cognitive testing for subsequent dynamic adaptation of the control unit during use. The information processing and output unit includes a module for capturing and identifying visual images, a headset module comprising built-in loudspeakers and a bone conduction function, and a module for identifying speech and sound signals.

Description

Communication system for people with hearing and visual impairment

The invention relates to special medical devices for individual use, intended for the rehabilitation of persons with disabilities, namely, the communication system of persons with impaired sensory functions of hearing and vision (deaf-deaf) to a neurocommunicator and can be used as a channel of communication between a person and the outside world.

The most acute problems for people with disabilities with impaired sensory functions of hearing and vision (deaf-deaf) are the lack of communication with people around and orientation in space. To solve these problems, devices are created that help deaf-blind people communicate, read, work on the Internet, navigate in space, and better perceive information by ear.

So, from the prior art there is known a device for the exchange of information between deaf-deaf and mute (RU 2197749, 01.27.2003, G09B 21/04, A61F 9/08), namely a means of communication between persons with hearing and visual impairment in cases where the exchange of information by known methods do not allow the implementation of external conditions. The device is easily interfaced with any transceivers (radiotelephone, regular telephone, etc.) and transmits information in the form of tactile sensations created by the electrical selection of the parameters of the electric effect, which exclude the interfering effect of the body's adaptation to electric exposure and polarization of the skin.

A known system for warning and communication of persons with special health needs (RU2303427, 07.27.2007, A61F 9/08). The system related to the technical means of rehabilitation of persons with disabilities has advanced functionality. The device contains a set of harmoniously working devices interconnected by bidirectional multi-bit (wireless) buses: wrist receiver, signal recognition device and a set of external signal sensors.

Known device GB 2302476 "One-handed speech and communication device for the speech-impaired or mute", consisting of two parts, one of which is in the user's hand and contains controls - a specialized keyboard, consisting of 5 buttons. The combination of the pressed buttons allows you to create 32 characters, which are then displayed on the display and voiced by a speech synthesizer. The device can be connected to a telephone, fax, computer and other devices through special connectors. In this case, not synthesized speech signals can be transmitted, but the corresponding digital codes. The second, more dimensional part of the device contains a power source, a speech synthesizer and a converter of the input code into characters on the display.

A mobile handheld device contains a power source; processor unit (control unit) with appropriate software; remote control with a group of buttons for the user to form the necessary code (keyboard block) blocks of wired interface with external devices (for connecting to telephone, sound recording equipment or a personal computer), LED indicator block, LED indicator coding block, speech synthesizer block, sound indication block, as well as memory blocks for LED indicator signals, prepared word and message codes , codes for transmitting digital information and a memory block of operation algorithms. The disadvantages of the known solutions include a limited range of tasks to be solved by the system and low application efficiency, non-ergonomic constructive separation of the device into two units connected by a cable, and the inability to wirelessly transmit information to external communication devices.

In part, these shortcomings have been resolved by a technical means for the rehabilitation of persons with disabilities (RU 2312646, 12.20.2007, A61F 9/08, A61F 11/04). The device generates the necessary voice messages and other signals that are situationally necessary for the exchange of information, and also recognizes sounds and speech. The device contains a set of coordinately working devices interconnected by bidirectional multi-bit (wireless) buses: an external device

(wireless) pairing with a telephone line, a mobile device for partial replacement of speech and hearing functions, a set of external information sensors. The device has advanced features of application, ergonomic and safe for the user, but does not solve all relevant technical and social problems.

The objective of the present invention is to develop a neurocommunicator aimed at ensuring the communication of persons with impaired sensory functions of hearing and vision (deaf-deaf), a safe, multi-functional, easy-to-use device with improved navigation qualities. The technical result achieved by using the present invention is to expand the ability to control the environment of deaf-blind users by increasing the functionality of the device. The present invention improves the independence and independence of the corresponding contingent of persons with disabilities in health.

The solution of the problem and the achievement of the specified technical result was made possible thanks to a combination of features of an independent claim of the claimed invention, the essence of which is that the proposed system is a block-modular structure connected wirelessly. The system includes three units: a control unit, a control unit, and an information processing and output unit. The control unit forms a microcomputer with a computing platform using cross-modal information conversion. The control unit is made in the form of a module for the functional testing of the deaf-blind, providing functional, psychological and cognitive tests for the subsequent dynamic adaptation of the control unit during use. And the information processing and output unit includes a module for capturing and recognizing visual images, a headset module with built-in speakers and a bone conduction function, and a module for recognizing speech and sound signals, a detailed description of which is presented below.

Further, in more detail, the invention is illustrated in the drawings, where

In FIG. 1 is a general view of a system diagram with the blocks and modules forming it;

FIG. 2 - the principle of cross-modal transcoding used in the system;

FIG. 3 is a general view of a module for capturing and recognizing visual images;

FIG. 4 is a three-dimensional sectional view of a module for capturing and recognizing visual images;

FIG. 5 is a two-dimensional sectional view of a module for capturing and recognizing visual images;

FIG. 6 is a diagram of the placement of indicators and buttons of a module for capturing and recognizing visual images;

7 is a three-dimensional view of an audio headset module with built-in speakers and a bone conduction function;

Fig - two-dimensional view of the module audio headset, without headphones;

Fig.9 is a two-dimensional view of the module audio headset in section;

Figure 10 is a General three-dimensional view of a speech recognition module;

11 is a speech recognition module, a top view;

Fig - module speech recognition, side view;

Fig - module speech recognition, side view in section.

The communication system for people with hearing and visual impairments (hereinafter referred to as the System) is a hardware-software complex consisting of a control unit 1, a control unit 2, and an information processing and output unit 3. The control unit 1 is a high-performance compact-sized microcomputer with a computing platform that uses cross-modal transcoding of information to a more sensitive blind channel or its sub-band. This block 1 is designed to receive incoming audio and visual information, distribute it among the modules of block 3, process it by recognizing images and objects, recognize speech and sounds, then cross-model transcode the information and output the information to the user in acoustic and / or graphic and / or tactile form with the possibility of control procedures. The cross-modal transcoding principle used in the system is shown in FIG. 2 and will be described below.

The control unit 2 is intended for functional testing of the deafblind and provides functional, psychological and cognitive tests for subsequent use of the 4 or 8 channel neurointerface for the dynamic adaptation of unit 1 during use.

Block 2 analyzes the electrical activity data of the cerebral cortex (EEG) of the user and, depending on the currently activated sensory modalities of the user, sends feedback to block 1. In this context of use, block 2 is a control device and allows you to evaluate the effectiveness of the cross-module information conversion process in the case of work with users who cannot independently confirm the fact that they receive valuable information from output devices.

The information processing and output unit 3 carries out the functions of capturing speech and sounds, transmitting acoustic signals and transmitting a video data stream, for which unit 3 includes three modules: a module for capturing and recognizing visual images 4, a headset module 5 with built-in speakers and a bone conduction function, and a module recognition of speech and sound signals 6. Modules 4,5,6 of block 3 and also block 2 are connected to block 1 wirelessly via Bluetooth or LTE (Long Term Evolution) and Wi-Fi connections.

The module for capturing and recognizing visual images (UT) 4 provides the ability to receive and transmit video information, its preliminary processing and subsequent feedback (interpretation of images, signaling about the location of an object, etc.). For a more objective and informative assessment of the environment of the world picture analysis, the system uses a signal from a video camera or several video cameras, as well as an ultrasonic range finder. This signal is automatically classified by image processing in module 4 and block 1.

The UT 4 module is made with the function of informing the user in sound and tactile form about the categories (classes) of recognized objects, the direction and distance to them from module 4, identified by people by the sound recorded by the user, the recognized printed text, about signs, pictograms and road signs for pedestrians.

The Braille display can be connected to the UT 4 module wirelessly or wired to output recognized text. Using the controls on the Braille display, you can scroll and scroll through the entire recognized list of objects.

As shown in FIG. 3-6, the module for capturing and recognizing visual images 4 is a device case, including a radiator 7, control buttons 8, status indicators 9, an ultrasonic range finder 10, a diode flashlight 11, a stereo pair of cameras 12, a microphone 13, an audio capture board 14, a control and power board 15, battery balancing board 16, strap holder 17, batteries 18, micro USB 19, USB 20, HDMI 21, charging 22, audio output 3.5 minijack 23, 24 on / off button, flashlight and rangefinder 25 board , button board 26, vibration motor 27, 28 — plug of the connectors. In FIG. Fig. 6 shows the device control buttons: object recognition button 29, text recognition button 30, plate recognition button 31, lamp on / off button 32, button for adding new faces 33, traffic sign recognition button 34, Search button 35. If you press briefly on the “Search” button 35 turns on and off the ultrasonic vibro-range finder to determine the distance to objects, and with a long press of the button 35 for a duration of more than 2 seconds, devices located at a distance of no more than 10 m from the UT 4 module are searched. UT 4 search commands the device’s body begins to vibrate.

The headset module (AG) 5 with built-in speakers and bone conduction function is presented from different angles in Fig.7-9. The main purpose of the device is the transmission of acoustic signals, taking into account the conductive hearing loss of the deaf-blind. In this case, for deaf users use headphones with combined air and bone conduction, and for users with partial hearing loss - headphones with the function of outputting information through air conduction and by transmitting sounds through the bones of the skull. The device provides signal reception via Bluetooth or from a microphone. The buttons on the body provide control over the volume of bone / air conduction and are responsible for selecting the signal source / output method / turning the headset on and off. The headset module 5 with built-in speakers and bone conduction function consists of a headband containing a flexible insert 39, 36 - control and mode buttons, 37 - air conduction volume control buttons, 38 - bone conduction volume control buttons, 40 - audio processing board for easy use 41 - controller board and Bluetooth, 42 - bone-conduction headphone, 43 - air-conduction headphone, 44 - battery, 45 - power cable, 46 - connection cable, 47 - air and bone conduction output, 48 - induction loop input.

The length of the wire for connecting headphones is 25 cm. The device’s operating time is at least 19 hours. Headphones with combined air and bone (sound transmission through the bones of the skull) conductivity allow the use of the system, including for deaf users.

The recognition module for speech and sound signals (MP) 6 and their presentation in text or tactile form is a device that, using directional microphones, captures speech or other sound information and translates it into text form with further presentation on paired devices for tactile output of information . In order to receive an acoustic signal from external sources, an array of microphones was used with a larger signal-to-noise ratio, the arrangement of which provides the direction of receiving an acoustic signal, which can be changed by an electronic device by phase shifts of the electric signal coming from each microphone separately to achieve the maximum ratio signal / noise.

MP 6 module (Figs. 10-13) includes 49 - a case, 50 - with a microphone array with sound processing, 51 - and an LED indicating the direction, 52 - a single-board computer located inside and performing analysis of incoming data, 53 - a battery, 54 - bracket, 55 - optical fiber connected to the LED board 51, 56 - acoustic Audiocore fabric, making the sound cleaner, suppressing noise, 57 - four clips holding the cover with glass 58, to ensure the visibility of the LED backlight; Inputs for any output devices: 59 - HDMI, for example, for connecting an external display, 60 - USB connector for connecting a Braille display by wire, 61 - LAN connector for wired Internet connection and 62 - micro-USB connector for charging the battery, and 63 - a two-position switch and 64 - a stereo jack for connecting a headset, 65 - The bottom to which the whole structure is mounted with reliable fasteners; 66 - buttons provide device control, and 67 - the array driver of microphones ensures uninterrupted operation of the microphone array. The developed device is powered by both the built-in battery and a standard power supply unit with an output voltage of 5V, connected to a household power supply network with a voltage of 220 V and a frequency of 50 Hz.

To display text information, any screen with an HDMI connection and / or a braille display can be used.

The device’s interface displays a list of all recognized phrases, direction to the speaker (left, right, right, back), battery level, microphone sensitivity level, status of presence or absence of an active sound source at the moment.

The implementation of the voice activity detector is as follows:

- conversion of the audio signal into the frequency domain;

- suppression of high-frequency noise in the signal;

- suppression of noise in the captured signal in the low frequency region;

- analysis of the purified audio signal for speech.

Speech recognition can be implemented using any cloud speech recognition platform.

For recognition of visual images and acoustic signals coming from the camera and an array of microphones, as well as for wireless access to the device using a Wi-Fi network, the system contains a single electronic signal processing device that runs on the Linux operating system.

To analyze the audio stream, a combination of digital signal processing and deep learning methods is used.

The system is used as follows. Turn on all devices by pressing the start button located on the main device of unit 1. All electronic modules 4,5,6 turn on automatically. Using block 1, the user can select a limited number of modules for use. Block 1 also contains individual settings and cross-modal transcoding parameters for each user. information, depending on user violations and the preferred channel for obtaining information. As shown in FIG. 2 The system allows you to work with input information received through the audio (sound) and visual channels. Modules 5 and 6 are responsible for processing the audio information. Module 4 is responsible for processing the visual information. After processing the input information in modules 4, 5, 6, using the built-in sensors, information is transmitted to block 1, in which cross-modal transcoding of information to more secure the user has touch channels. Block 1 transmits information to modules 4, 5, 6, which then provide the output of the converted information in visual (text), audio (sounds) and tactile (vibration and embossed font) forms, this information is received by the user, a person with impaired hearing and vision ( deafblind). During the operation of the System, the MFT 3 unit is used to register the user's EEG and evaluate the level of efficiency of use of the selected sensory modalities; these data affect the values of the parameters of the cross-modal information conversion process.

Claims

Claim

1. A communication system for persons with hearing and visual impairment, which is a hardware-software complex with a block-modular structure interconnected wirelessly, characterized in that it includes

- a control unit (1) represented by a microcomputer with a computing platform using cross-modal information conversion;

- the control unit (2) in the form of a module for the functional testing of the deaf-blind, providing functional, psychological and cognitive tests using the software installed in unit 1 for subsequent dynamic adaptation of unit 1 during use;

- information processing and output unit (3), configured to capture speech and sounds, transmit acoustic signals and transmit video data stream, for which unit (3) includes a module for capturing and recognizing visual images (4), headset module (5) with built-in loudspeakers and bone conduction function and a recognition module for speech and sound signals (6), while

- the module for capturing and recognizing visual images (4) is a body of a device equipped with an ultrasonic rangefinder (10), a diode flashlight (11), a stereo pair of cameras (12), a microphone (13), audio capture cards (14), control and power supply (15 ), balancing the battery (16), flashlight and rangefinder (25), vibration motor (27) and battery (18),

- a headset module (5) with built-in speakers and a bone conduction function, includes a headband, an air conduction earpiece (43), a bone conduction earpiece (42), as well as audio processing boards (40), a controller and Bluetooth (41) and batteries (44) the recognition module for speech and sound signals (6), contains a housing (49) with a microphone for sound processing (50) and an LED direction indicator (51), a single-board computer (52), and a battery (53) while the location of the directional microphones provides directional reception acoustic signal.

2. The communication system for persons with hearing and visual impairment according to claim 1, excommunicated by the fact that it is equipped with a power supply.

3. The communication system for people with hearing and visual impairment according to claim 1, characterized in that the Braille display is connected to the module for capturing and recognizing visual images wirelessly or wired.