WO2020117089A1 - Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue - Google Patents

Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue Download PDF

Info

Publication number
WO2020117089A1
WO2020117089A1 PCT/RU2018/000794 RU2018000794W WO2020117089A1 WO 2020117089 A1 WO2020117089 A1 WO 2020117089A1 RU 2018000794 W RU2018000794 W RU 2018000794W WO 2020117089 A1 WO2020117089 A1 WO 2020117089A1
Authority
WO
WIPO (PCT)
Prior art keywords
module
persons
unit
hearing
speech
Prior art date
Application number
PCT/RU2018/000794
Other languages
English (en)
Russian (ru)
Inventor
Денис Сергеевич КУЛЕШОВ
Андрей Павлович БЫТЕЙЩИКОВ
Александр Валерьевич ГОРЮНОВ
Марк Викторович ЗОЛОТАРЁВ
Николай Николаевич КИРИЧЕНКО
Евгений Константинович ПОПОВ
Александр Николаевич СЕМЕНЕНКО
Александр Викторович Попов
Original Assignee
Автономная Некоммерческая Организация "Научно-Производственная Лаборатория "Сенсорные Технологии Для Слепоглухих"
Общество с ограниченной ответственностью "Нейроботикс"
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Автономная Некоммерческая Организация "Научно-Производственная Лаборатория "Сенсорные Технологии Для Слепоглухих", Общество с ограниченной ответственностью "Нейроботикс" filed Critical Автономная Некоммерческая Организация "Научно-Производственная Лаборатория "Сенсорные Технологии Для Слепоглухих"
Priority to PCT/RU2018/000794 priority Critical patent/WO2020117089A1/fr
Publication of WO2020117089A1 publication Critical patent/WO2020117089A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61F—FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
    • A61F11/00—Methods or devices for treatment of the ears or hearing sense; Non-electric hearing aids; Methods or devices for enabling ear patients to achieve auditory perception through physiological senses other than hearing sense; Protective devices for the ears, carried on the body or in the hand
    • A61F11/04—Methods or devices for enabling ear patients to achieve auditory perception through physiological senses other than hearing sense, e.g. through the touch sense
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61F—FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
    • A61F9/00—Methods or devices for treatment of the eyes; Devices for putting-in contact lenses; Devices to correct squinting; Apparatus to guide the blind; Protective devices for the eyes, carried on the body or in the hand
    • A61F9/08—Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids

Definitions

  • the invention relates to special medical devices for individual use, intended for the rehabilitation of persons with disabilities, namely, the communication system of persons with impaired sensory functions of hearing and vision (deaf-deaf) to a neurocommunicator and can be used as a channel of communication between a person and the outside world.
  • deaf-deaf The most acute problems for people with disabilities with impaired sensory functions of hearing and vision (deaf-deaf) are the lack of communication with people around and orientation in space. To solve these problems, devices are created that help deaf-blind people communicate, read, work on the Internet, navigate in space, and better perceive information by ear.
  • a device for the exchange of information between deaf-deaf and mute (RU 2197749, 01.27.2003, G09B 21/04, A61F 9/08), namely a means of communication between persons with hearing and visual impairment in cases where the exchange of information by known methods do not allow the implementation of external conditions.
  • the device is easily interfaced with any transceivers (radiotelephone, regular telephone, etc.) and transmits information in the form of tactile sensations created by the electrical selection of the parameters of the electric effect, which exclude the interfering effect of the body's adaptation to electric exposure and polarization of the skin.
  • the device contains a set of harmoniously working devices interconnected by bidirectional multi-bit (wireless) buses: wrist receiver, signal recognition device and a set of external signal sensors.
  • Known device GB 2302476 "One-handed speech and communication device for the speech-impaired or mute", consisting of two parts, one of which is in the user's hand and contains controls - a specialized keyboard, consisting of 5 buttons. The combination of the pressed buttons allows you to create 32 characters, which are then displayed on the display and voiced by a speech synthesizer.
  • the device can be connected to a telephone, fax, computer and other devices through special connectors. In this case, not synthesized speech signals can be transmitted, but the corresponding digital codes.
  • the second, more dimensional part of the device contains a power source, a speech synthesizer and a converter of the input code into characters on the display.
  • a mobile handheld device contains a power source; processor unit (control unit) with appropriate software; remote control with a group of buttons for the user to form the necessary code (keyboard block) blocks of wired interface with external devices (for connecting to telephone, sound recording equipment or a personal computer), LED indicator block, LED indicator coding block, speech synthesizer block, sound indication block, as well as memory blocks for LED indicator signals, prepared word and message codes , codes for transmitting digital information and a memory block of operation algorithms.
  • the disadvantages of the known solutions include a limited range of tasks to be solved by the system and low application efficiency, non-ergonomic constructive separation of the device into two units connected by a cable, and the inability to wirelessly transmit information to external communication devices.
  • the device generates the necessary voice messages and other signals that are situationally necessary for the exchange of information, and also recognizes sounds and speech.
  • the device contains a set of coordinately working devices interconnected by bidirectional multi-bit (wireless) buses: an external device
  • the device has advanced features of application, ergonomic and safe for the user, but does not solve all relevant technical and social problems.
  • the objective of the present invention is to develop a neurocommunicator aimed at ensuring the communication of persons with impaired sensory functions of hearing and vision (deaf-deaf), a safe, multi-functional, easy-to-use device with improved navigation qualities.
  • the technical result achieved by using the present invention is to expand the ability to control the environment of deaf-blind users by increasing the functionality of the device.
  • the present invention improves the independence and independence of the corresponding contingent of persons with disabilities in health.
  • the system includes three units: a control unit, a control unit, and an information processing and output unit.
  • the control unit forms a microcomputer with a computing platform using cross-modal information conversion.
  • the control unit is made in the form of a module for the functional testing of the deaf-blind, providing functional, psychological and cognitive tests for the subsequent dynamic adaptation of the control unit during use.
  • the information processing and output unit includes a module for capturing and recognizing visual images, a headset module with built-in speakers and a bone conduction function, and a module for recognizing speech and sound signals, a detailed description of which is presented below.
  • FIG. 1 is a general view of a system diagram with the blocks and modules forming it;
  • FIG. 2 the principle of cross-modal transcoding used in the system
  • FIG. 3 is a general view of a module for capturing and recognizing visual images
  • FIG. 4 is a three-dimensional sectional view of a module for capturing and recognizing visual images
  • FIG. 5 is a two-dimensional sectional view of a module for capturing and recognizing visual images
  • FIG. 6 is a diagram of the placement of indicators and buttons of a module for capturing and recognizing visual images
  • FIG. 7 is a three-dimensional view of an audio headset module with built-in speakers and a bone conduction function
  • Fig.9 is a two-dimensional view of the module audio headset in section
  • Figure 10 is a General three-dimensional view of a speech recognition module
  • 11 is a speech recognition module, a top view
  • the communication system for people with hearing and visual impairments (hereinafter referred to as the System) is a hardware-software complex consisting of a control unit 1, a control unit 2, and an information processing and output unit 3.
  • the control unit 1 is a high-performance compact-sized microcomputer with a computing platform that uses cross-modal transcoding of information to a more sensitive blind channel or its sub-band.
  • This block 1 is designed to receive incoming audio and visual information, distribute it among the modules of block 3, process it by recognizing images and objects, recognize speech and sounds, then cross-model transcode the information and output the information to the user in acoustic and / or graphic and / or tactile form with the possibility of control procedures.
  • the cross-modal transcoding principle used in the system is shown in FIG. 2 and will be described below.
  • the control unit 2 is intended for functional testing of the deafblind and provides functional, psychological and cognitive tests for subsequent use of the 4 or 8 channel neurointerface for the dynamic adaptation of unit 1 during use.
  • Block 2 analyzes the electrical activity data of the cerebral cortex (EEG) of the user and, depending on the currently activated sensory modalities of the user, sends feedback to block 1.
  • EEG cerebral cortex
  • block 2 is a control device and allows you to evaluate the effectiveness of the cross-module information conversion process in the case of work with users who cannot independently confirm the fact that they receive valuable information from output devices.
  • the information processing and output unit 3 carries out the functions of capturing speech and sounds, transmitting acoustic signals and transmitting a video data stream, for which unit 3 includes three modules: a module for capturing and recognizing visual images 4, a headset module 5 with built-in speakers and a bone conduction function, and a module recognition of speech and sound signals 6.
  • Modules 4,5,6 of block 3 and also block 2 are connected to block 1 wirelessly via Bluetooth or LTE (Long Term Evolution) and Wi-Fi connections.
  • the module for capturing and recognizing visual images (UT) 4 provides the ability to receive and transmit video information, its preliminary processing and subsequent feedback (interpretation of images, signaling about the location of an object, etc.). For a more objective and informative assessment of the environment of the world picture analysis, the system uses a signal from a video camera or several video cameras, as well as an ultrasonic range finder. This signal is automatically classified by image processing in module 4 and block 1.
  • the UT 4 module is made with the function of informing the user in sound and tactile form about the categories (classes) of recognized objects, the direction and distance to them from module 4, identified by people by the sound recorded by the user, the recognized printed text, about signs, pictograms and road signs for pedestrians.
  • the Braille display can be connected to the UT 4 module wirelessly or wired to output recognized text. Using the controls on the Braille display, you can scroll and scroll through the entire recognized list of objects.
  • the module for capturing and recognizing visual images 4 is a device case, including a radiator 7, control buttons 8, status indicators 9, an ultrasonic range finder 10, a diode flashlight 11, a stereo pair of cameras 12, a microphone 13, an audio capture board 14, a control and power board 15, battery balancing board 16, strap holder 17, batteries 18, micro USB 19, USB 20, HDMI 21, charging 22, audio output 3.5 minijack 23, 24 on / off button, flashlight and rangefinder 25 board , button board 26, vibration motor 27, 28 — plug of the connectors.
  • FIG. Fig. 6 shows the device control buttons: object recognition button 29, text recognition button 30, plate recognition button 31, lamp on / off button 32, button for adding new faces 33, traffic sign recognition button 34, Search button 35.
  • the headset module (AG) 5 with built-in speakers and bone conduction function is presented from different angles in Fig.7-9.
  • the main purpose of the device is the transmission of acoustic signals, taking into account the conductive hearing loss of the deaf-blind.
  • the device provides signal reception via Bluetooth or from a microphone.
  • the buttons on the body provide control over the volume of bone / air conduction and are responsible for selecting the signal source / output method / turning the headset on and off.
  • the headset module 5 with built-in speakers and bone conduction function consists of a headband containing a flexible insert 39, 36 - control and mode buttons, 37 - air conduction volume control buttons, 38 - bone conduction volume control buttons, 40 - audio processing board for easy use 41 - controller board and Bluetooth, 42 - bone-conduction headphone, 43 - air-conduction headphone, 44 - battery, 45 - power cable, 46 - connection cable, 47 - air and bone conduction output, 48 - induction loop input.
  • the length of the wire for connecting headphones is 25 cm.
  • the device’s operating time is at least 19 hours.
  • Headphones with combined air and bone (sound transmission through the bones of the skull) conductivity allow the use of the system, including for deaf users.
  • the recognition module for speech and sound signals (MP) 6 and their presentation in text or tactile form is a device that, using directional microphones, captures speech or other sound information and translates it into text form with further presentation on paired devices for tactile output of information .
  • an array of microphones was used with a larger signal-to-noise ratio, the arrangement of which provides the direction of receiving an acoustic signal, which can be changed by an electronic device by phase shifts of the electric signal coming from each microphone separately to achieve the maximum ratio signal / noise.
  • MP 6 module (Figs. 10-13) includes 49 - a case, 50 - with a microphone array with sound processing, 51 - and an LED indicating the direction, 52 - a single-board computer located inside and performing analysis of incoming data, 53 - a battery, 54 - bracket, 55 - optical fiber connected to the LED board 51, 56 - acoustic Audiocore fabric, making the sound cleaner, suppressing noise, 57 - four clips holding the cover with glass 58, to ensure the visibility of the LED backlight; Inputs for any output devices: 59 - HDMI, for example, for connecting an external display, 60 - USB connector for connecting a Braille display by wire, 61 - LAN connector for wired Internet connection and 62 - micro-USB connector for charging the battery, and 63 - a two-position switch and 64 - a stereo jack for connecting a headset, 65 - The bottom to which the whole structure is mounted with reliable fasteners; 66 - buttons provide device control, and 67 - the array
  • any screen with an HDMI connection and / or a braille display can be used.
  • the device’s interface displays a list of all recognized phrases, direction to the speaker (left, right, right, back), battery level, microphone sensitivity level, status of presence or absence of an active sound source at the moment.
  • the implementation of the voice activity detector is as follows:
  • Speech recognition can be implemented using any cloud speech recognition platform.
  • the system For recognition of visual images and acoustic signals coming from the camera and an array of microphones, as well as for wireless access to the device using a Wi-Fi network, the system contains a single electronic signal processing device that runs on the Linux operating system.
  • Block 1 also contains individual settings and cross-modal transcoding parameters for each user. information, depending on user violations and the preferred channel for obtaining information. As shown in FIG. 2 The system allows you to work with input information received through the audio (sound) and visual channels. Modules 5 and 6 are responsible for processing the audio information. Module 4 is responsible for processing the visual information. After processing the input information in modules 4, 5, 6, using the built-in sensors, information is transmitted to block 1, in which cross-modal transcoding of information to more secure the user has touch channels.
  • Block 1 transmits information to modules 4, 5, 6, which then provide the output of the converted information in visual (text), audio (sounds) and tactile (vibration and embossed font) forms, this information is received by the user, a person with impaired hearing and vision ( deafblind).
  • the MFT 3 unit is used to register the user's EEG and evaluate the level of efficiency of use of the selected sensory modalities; these data affect the values of the parameters of the cross-modal information conversion process.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Animal Behavior & Ethology (AREA)
  • Vascular Medicine (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Neurology (AREA)
  • Biophysics (AREA)
  • Otolaryngology (AREA)
  • Psychology (AREA)
  • Multimedia (AREA)
  • Physiology (AREA)
  • Signal Processing (AREA)
  • Quality & Reliability (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Ophthalmology & Optometry (AREA)
  • Rehabilitation Tools (AREA)

Abstract

L'invention concerne des dispositifs médicaux spéciaux à usage individuel servant au rétablissement de personnes handicapées, et concerne notamment un système de communication pour personnes présentant des toubles des fonctions sensorielles de l'ouïe et de la vue (aveugles-sourds) qui consiste en un neurocommunicateur, et qui peut être utilisé comme canal de communication d'une personne avec le monde extérieur. L'invention a pour but d'élargir les capacités de contrôle sur le milieu environnant d'utilisateurs aveugles et sourds, et permet d'augmenter l'indépendance et la liberté d'un contingent de personnes ayant des capacités limités en termes de santé, grâce à une augmentation de la fonctionnalité du dispositif. Ce système consiste en une structure d'unités modulaire connectée selon un procédé sans fil. Le système comprend trois unités: une unité de commande, une unité de contrôle et une unité de traitement et de sortie d'informations. L'unité de commande consiste en un microordinateur avec une plateforme de calcul qui utilise un transcodage intermodal des informations. L'unité de contrôle se présente sous forme d'un module de test fonctionnel des aveugles-sourds, qui permet de réaliser des tests fonctionnels psychologiques et cognitifs, en vue de l'adaptation dynamique ultérieure de l'unité de commande lors du processus d'utilisation. L'unité de traitement et de sortie d'informations comprend un module de saisie et de reconnaissance de modèles visuels, un module de casque intégré avec des haut-parleurs intégrés et une fonction de conductivité osseuse, et un module de reconnaissance de parole et de signaux sonores.
PCT/RU2018/000794 2018-12-07 2018-12-07 Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue WO2020117089A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/RU2018/000794 WO2020117089A1 (fr) 2018-12-07 2018-12-07 Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/RU2018/000794 WO2020117089A1 (fr) 2018-12-07 2018-12-07 Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue

Publications (1)

Publication Number Publication Date
WO2020117089A1 true WO2020117089A1 (fr) 2020-06-11

Family

ID=70974379

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2018/000794 WO2020117089A1 (fr) 2018-12-07 2018-12-07 Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue

Country Status (1)

Country Link
WO (1) WO2020117089A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4982434A (en) * 1989-05-30 1991-01-01 Center For Innovative Technology Supersonic bone conduction hearing aid and method
RU2312646C2 (ru) * 2005-12-05 2007-12-20 Андрей Всеволодович Воробьев Устройство частичной замены функций речи и слуха
US20160255270A1 (en) * 2014-05-09 2016-09-01 Boe Technology Group Co., Ltd. Guide Spectacle

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4982434A (en) * 1989-05-30 1991-01-01 Center For Innovative Technology Supersonic bone conduction hearing aid and method
RU2312646C2 (ru) * 2005-12-05 2007-12-20 Андрей Всеволодович Воробьев Устройство частичной замены функций речи и слуха
US20160255270A1 (en) * 2014-05-09 2016-09-01 Boe Technology Group Co., Ltd. Guide Spectacle

Similar Documents

Publication Publication Date Title
US9095423B2 (en) Apparatus and method for providing failed-attempt feedback using a camera on glasses
US6240392B1 (en) Communication device and method for deaf and mute persons
US20130343585A1 (en) Multisensor hearing assist device for health
US9781524B2 (en) Communication system
TW201637458A (zh) 利用電磁型振動單元之軟骨傳導聽取裝置、及電磁型振動單元
CN104983511A (zh) 针对全盲视觉障碍者的语音帮助智能眼镜系统
KR102047988B1 (ko) 시력 취약계층을 위한 시력 보조장치와 원격관리장치 및 시력 보조방법
JP4445178B2 (ja) 通信システムおよびそのための適切な制御ユニット
US20230091607A1 (en) Psychoacoustics-based audio encoding method and apparatus
US20200227067A1 (en) Communication aid system
KR101354927B1 (ko) 청각장애인을 위한 머리 착용형 디스플레이장치
CN111065020A (zh) 音频数据处理的方法和装置
KR100748432B1 (ko) 청각 장애인용 착용형 단말 장치
CN112002186B (zh) 一种基于增强现实技术的信息无障碍系统及方法
RU198673U1 (ru) Портативное устройство распознавания речи и звуковых сигналов
KR100806639B1 (ko) 청각장애 및 난청인용 상황별 인식장치
WO2020117089A1 (fr) Système de communication pour personnes souffrant de troubles de l'ouïe et de la vue
RU2303427C2 (ru) Система для оповещения и коммуникации лиц со специальными потребностями по здоровью
CN111128180A (zh) 一种听力障碍者的辅助对话系统
CN110031976A (zh) 一种具有报警功能的眼镜及其控制方法
CN110087162A (zh) 一种骨传导降噪通信方法及通信耳机
CN105641900B (zh) 一种呼吸状态提醒方法和电子设备以及系统
JPH10282985A (ja) 聴覚支援装置
KR20160013657A (ko) 청각장애인을 위한 손목시계
RU182005U1 (ru) Приёмник для лиц с нарушением функций слуха и речи

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18942167

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 05/10/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18942167

Country of ref document: EP

Kind code of ref document: A1