CN112183216A - Auxiliary system for communication of disabled people - Google Patents
Auxiliary system for communication of disabled people Download PDFInfo
- Publication number
- CN112183216A CN112183216A CN202010910889.2A CN202010910889A CN112183216A CN 112183216 A CN112183216 A CN 112183216A CN 202010910889 A CN202010910889 A CN 202010910889A CN 112183216 A CN112183216 A CN 112183216A
- Authority
- CN
- China
- Prior art keywords
- module
- alarm
- sign language
- information
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004891 communication Methods 0.000 title claims abstract description 27
- 238000006243 chemical reaction Methods 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 23
- 230000002452 interceptive effect Effects 0.000 claims abstract description 22
- 238000012549 training Methods 0.000 claims abstract description 16
- 206010011878 Deafness Diseases 0.000 claims abstract description 10
- 230000036541 health Effects 0.000 claims abstract description 10
- 238000007726 management method Methods 0.000 claims description 15
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 230000003068 static effect Effects 0.000 claims description 12
- 238000012937 correction Methods 0.000 claims description 9
- 239000003205 fragrance Substances 0.000 claims description 9
- WHXSMMKQMYFTQS-UHFFFAOYSA-N Lithium Chemical compound [Li] WHXSMMKQMYFTQS-UHFFFAOYSA-N 0.000 claims description 6
- 238000013500 data storage Methods 0.000 claims description 6
- 238000011156 evaluation Methods 0.000 claims description 6
- 229910052744 lithium Inorganic materials 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000010438 heat treatment Methods 0.000 claims description 5
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 claims description 4
- 229910052802 copper Inorganic materials 0.000 claims description 4
- 239000010949 copper Substances 0.000 claims description 4
- 239000004973 liquid crystal related substance Substances 0.000 claims description 4
- 238000000034 method Methods 0.000 claims description 4
- 230000009471 action Effects 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000003709 image segmentation Methods 0.000 claims description 3
- 230000000877 morphologic effect Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000004080 punching Methods 0.000 claims description 3
- 239000007787 solid Substances 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 2
- 239000000872 buffer Substances 0.000 claims 1
- 230000003139 buffering effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 4
- 208000032041 Hearing impaired Diseases 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000005611 electricity Effects 0.000 description 2
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000000741 silica gel Substances 0.000 description 1
- 229910002027 silica gel Inorganic materials 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9532—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration by the use of local operators
- G06T5/30—Erosion or dilatation, e.g. thinning
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/26—Speech to text systems
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
Abstract
The invention provides an auxiliary system for communication of disabled people, which comprises: the medical treatment system comprises a sign language identification unit, an intelligent alarm unit, an event record reminding unit, an information conversion processing unit, a medical treatment auxiliary unit and an interactive display unit; the sign language recognition unit is used for inputting hand posture information of the deaf-mute; the intelligent alarm unit is used for alarming events and comprises an alarm priority setting module, an alarm information caching processing module, an alarm mode selection module, an online alarm module and an offline alarm module; the event recording reminding unit is used for recording the items required to be completed by the user within a period of time and carrying out timing reminding; the medical auxiliary unit is used for verifying whether the identity of the user is legal, storing the case and the rechecking report of the user and outputting the health report of the user, carrying out travel navigation on the user and assisting the user in carrying out rehabilitation training.
Description
Technical Field
The invention relates to the technical field of intelligent communication auxiliary equipment, in particular to an auxiliary system for communication of disabled people.
Background
For the hearing-impaired people and the language-impaired people, the sign language is a main communication tool, and because the common people have low understanding degree of the sign language, the communication obstacle between the hearing-impaired people and other people is directly caused, so that a lot of inconvenience is brought to the life and the safety of the hearing-impaired people, and the living space and the development space of the deaf-mute are limited. How to study an intelligent auxiliary system which can help deaf-mute to smoothly express their thoughts and alleviate inconvenience in life and can help deaf-mute to communicate and communicate is very significant, along with the improvement of technological level, some of intelligent technologies such as a lot of wearable, AI projection, intelligent recognition and voice control have been applied to real work and life, and the intelligent technologies can also well assist people with language communication obstacles to simply communicate with the outside, but because the technologies need powerful computing power and high-end application equipment, the cost is increased, the complex function also increases the burden of operation, and the recognition precision is not high, the communication is caused to be problematic.
In conclusion, the auxiliary system for communication of the disabled people, which is simple and convenient to operate, high in safety, capable of meeting the daily simple communication requirements of the deaf-mute and improving the use experience of the user, is a problem which needs to be solved urgently by technical personnel in the field.
Disclosure of Invention
In view of the above-mentioned problems and needs, the present invention provides an assistance system for communication of disabled people, which can solve the above technical problems by adopting the following technical solutions.
In order to achieve the purpose, the invention provides the following technical scheme: an assistance system for communication of disabled persons, comprising: the medical treatment system comprises a sign language identification unit, an intelligent alarm unit, an event record reminding unit, an information conversion processing unit, a medical treatment auxiliary unit and an interactive display unit;
the sign language recognition unit is used for inputting hand posture information of the deaf-mute and sending a recognition result to the voice playing unit for voice playing, the sign language recognition unit comprises a sign language information input module, a sign language information recognition module and a correction output module, and the sign language information input module, the sign language information recognition module and the correction output module are sequentially connected;
the intelligent alarm unit is used for event alarm and comprises an alarm priority setting module, an alarm information caching processing module, an alarm mode selecting module, an online alarm module and an offline alarm module, wherein the alarm mode comprises an active alarm mode and an automatic alarm mode, the active alarm mode is used for autonomous alarm of a user, the automatic alarm mode alarms according to a set threshold value, and when the alarm information caching processing module detects that a related parameter value exceeds the alarm threshold value, alarm information is sent to the online alarm module and the offline alarm module according to the priority level set by the alarm priority setting module;
the event record reminding unit is used for recording the items required to be completed by the user within a period of time and carrying out timed reminding, and comprises an information reminding module and a time recording module;
the information conversion processing unit comprises a voice playing module, a voice text conversion module, a text image conversion module, a database storage module, a cloud server and a user information association list, wherein the voice playing module is used for playing sign language recognition results, the voice playing module comprises a TTS voice chip, a loudspeaker, a voice mode selection module and a volume adjustment module, the voice playing module is connected with the correction output module, the voice text conversion module, the text image conversion module and the interactive display unit are sequentially connected, and the database storage module comprises a sign language data storage database, a user information storage database, an event storage database and an alarm record storage database;
the medical auxiliary unit comprises a user identity verification module, a health evaluation module, an intelligent navigation positioning module and a rehabilitation training module, wherein the user identity verification module is used for verifying whether the identity of a user is legal or not, the health evaluation module is used for storing a case and a rechecking report of the user and outputting the health report of the user, the intelligent navigation positioning module is used for the user to conduct travel navigation, the rehabilitation training module comprises a rehabilitation training video storage module and a user card punching module, and the rehabilitation training module is used for the user to conduct rehabilitation training.
Preferably, the interactive display unit comprises a liquid crystal interactive display, and the interactive display unit is used for displaying text information, sign language information and projection related video information corresponding to the voice expression content of the communicant.
More preferably, the information conversion processing unit further includes a sign language action 3D reconstruction module, and the 3D reconstruction module is configured to perform 3D reconstruction on the sign language image converted by the text image conversion module and restore a dynamic 3D sign language video through the interactive display unit.
More preferably, still include wireless mobile terminal and intelligent pronunciation bracelet, mutual display element all sets up on the wireless mobile terminal, still be provided with the miniature pickup microphone that is used for typeeing communication person's voice information on the intelligent pronunciation bracelet and be used for controlling the control switch that miniature pickup microphone opened and close, miniature pickup microphone with control switch with voice text conversion module is connected.
Preferably, still include the power supply management unit, the power supply management unit includes that low-voltage reminds module, battery management module, lithium cell, the module is reminded including the battery electric quantity board to low-voltage, battery management module includes the PMU chip, the module is reminded to low-voltage with the lithium cell all with the battery management module electricity is connected.
Preferably, after the intelligent alarm unit receives alarm information, the intelligent alarm unit firstly sends the alarm information caching value to the alarm information caching processing module, and when another piece of alarm information is received again in one section, the intelligent alarm unit sends the alarm information in sequence according to the default priority of the alarm priority setting module, otherwise, the intelligent alarm unit directly sends the alarm information to the online alarm module and the offline alarm module, wherein the online alarm module comprises a GSM automatic dialing module, and the offline alarm module comprises an acousto-optic alarm circuit.
More preferably, sign language information entry module includes two mesh cameras and bracing piece, sign language information entry module sets up on the wireless mobile terminal, two mesh cameras set up on the bracing piece, the bracing piece card is put at ordinary times on the wireless mobile terminal, will during the use the bracing piece pulls open.
Preferably, the sign language information recognition module adopts a sign language recognition algorithm based on deep learning, and the sign language recognition algorithm comprises: the sign language information input module collects gesture images and videos to construct a static sign language data set and a sign language video set, and the static sign language data set and the sign language video set are stored in the sign language data storage database; preprocessing and segmenting the depth image in the static sign language data set to obtain a target sign part, and obtaining a target sign part according to a formula K-cMγStretching the image contrast by adopting a power law conversion method to realize image contrast enhancement, wherein c and gamma are normal numbers, and the value ranges of K and M are [0, 1%]Performing morphological processing by adopting closed operation, and removing noise in the image by adopting Gaussian filtering to obtain a clear gesture image segmentation result; taking the gesture image as input, and adopting a CNN convolutional neural network to perform static isolated word sign language recognition; and extracting the gesture features of the video sequence in the sign language video set by using a CNN convolutional neural network, taking the extracted feature sequence as the input of an LSTM network, identifying the obtained feature sequence through the LSTM network, and outputting a sign language identification result to obtain a gesture class label corresponding to each of N features of the feature vector of the video sequence.
More preferably, the information reminding module comprises a vibration module, a fragrance volatilization module and a light module, wherein the vibration module, the fragrance volatilization module and the light module are all arranged in the intelligent voice bracelet, and the fragrance volatilization module comprises a heating copper sheet and a solid fragrance.
Preferably, the intelligent navigation positioning module comprises a gesture command input module and an electronic map module, a user can input a specific gesture command through the gesture command input module, the specific gesture command corresponds to a specific location query instruction, and the specific location query instruction is sent to the electronic map module to obtain a corresponding navigation route.
According to the technical scheme, the invention has the beneficial effects that: the method is simple and convenient to operate and high in safety, can meet the daily simple communication requirements of the deaf-mute, and improves the use experience of the user.
In addition to the above objects, features and advantages, preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings so that the features and advantages of the present invention can be easily understood.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly described, wherein the drawings are only used for illustrating some embodiments of the present invention and do not limit all embodiments of the present invention thereto.
Fig. 1 is a schematic structural diagram of an auxiliary system for communication of disabled persons according to the present invention.
Fig. 2 is a schematic structural diagram of the intelligent voice bracelet in the invention.
Fig. 3 is a schematic structural diagram of a wireless mobile terminal according to the present invention.
Fig. 4 is a schematic diagram illustrating specific steps of the sign language identification algorithm in this embodiment.
Reference numerals: the device comprises an interactive display unit 1, a sign language information input module 2, an offline alarm module 3, a voice playing module 4, a vibration module 5, a fragrance volatilization module 6, a light module 7 and a control switch 9 for starting and stopping a miniature pickup microphone 8.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of specific embodiments of the present invention. Like reference symbols in the various drawings indicate like elements. It should be noted that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention without any inventive step, are within the scope of protection of the invention.
The auxiliary system for communication of the disabled is simple and convenient to operate, high in safety, capable of meeting daily simple communication requirements of the deaf-mute and capable of improving use experience of the user. As shown in fig. 1 to 4, the system includes: the medical treatment system comprises a sign language recognition unit, an intelligent alarm unit, an event recording reminding unit, an information conversion processing unit, a medical treatment auxiliary unit and an interactive display unit 1, wherein the interactive display unit 1 comprises a liquid crystal interactive display, and the liquid crystal interactive display is used for displaying text information and 3D sign language information corresponding to voice expression contents of a communicator and projection-related rehabilitation videos or audio-video information.
In the system, the sign language recognition unit is used for inputting hand posture information of the deaf-mute and sending a recognition result to the voice playing unit for voice playing, the sign language recognition unit comprises a sign language information input module 2, a sign language information recognition module and a correction output module, and the sign language information input module 2, the sign language information recognition module and the correction output module are connected in sequence.
The intelligent alarm unit is used for alarming when being impacted or in emergency, the intelligent alarm unit comprises an alarm priority setting module, an alarm information caching processing module, an alarm mode selecting module, an online alarm module and an offline alarm module 3, the alarm mode comprises an active alarm mode and an automatic alarm mode, the active alarm mode is used for users to alarm autonomously through an alarm button, the automatic alarm mode alarms according to a set threshold value, and when the alarm information caching processing module detects that related parameter values exceed the alarm threshold value, the alarm priority setting module sets the priority level of the alarm priority setting module to the online alarm module and the offline alarm module 3 sends alarm information. After the intelligent alarm unit receives alarm information, the intelligent alarm unit firstly caches the alarm information to the alarm information caching processing module, receives another piece of alarm information again in one section, and then sequentially sends the alarm information according to the default priority of the alarm priority setting module, otherwise, the intelligent alarm unit directly sends the alarm information to the online alarm module and the offline alarm module 3, the online alarm module comprises a GSM automatic dialing module, and the offline alarm module 3 comprises an audible and visual alarm circuit.
The event record reminding unit is used for recording the events which need to be finished by the user in a period of time, such as one day or one week, and carrying out timing reminding, and comprises an information reminding module and a time recording module. Information warning module is including vibration module 5, fragrant atmosphere volatilization module 6 and light module 7, vibration module 5 fragrant atmosphere volatilization module 6 with light module 7 all sets up in the intelligence pronunciation bracelet, fragrant atmosphere volatilization module 6 is including heating copper sheet and solid fragrant atmosphere, heating copper sheet links to each other with heating circuit, heats automatic stop after 8 seconds, intelligence pronunciation bracelet adopts the silica gel material.
The information conversion processing unit comprises a voice playing module 4, a voice text conversion module, a text image conversion module, a database storage module, a cloud server and a user information association list, and further comprises a sign language action 3D reconstruction module, wherein the 3D reconstruction module is used for performing 3D reconstruction on the sign language image converted by the text image conversion module and restoring a dynamic 3D sign language video through the interactive display unit 1. The voice playing module 4 is used for playing sign language recognition results, the voice playing module 4 comprises a TTS voice chip, a loudspeaker, a voice mode selection module and a volume adjusting module, and the voice playing module 4 is connected with the correction output module and can play recognized sign language information in a voice mode to achieve the purpose of communicating with ordinary people. The voice text conversion module, the text image conversion module and the interactive display unit 1 are sequentially connected, the voice text conversion module can convert the speaking content of common people into text information, then the text information is converted into sign language images with corresponding semantics through the text image conversion module, then the sign language images are subjected to 3D reconstruction, dynamic 3D sign language videos are restored through the interactive display unit 1, the database storage module comprises a sign language data storage database, a user information storage database, an event storage database and an alarm record storage database, and the database storage module is connected with a cloud server.
The medical auxiliary unit in the system comprises a user identity verification module, a health evaluation module, an intelligent navigation positioning module and a rehabilitation training module, wherein the user identity verification module is used for verifying whether the identity of a user is legal or not, the health evaluation module is used for storing a case and a rechecking report of the user and outputting the health report of the user, the intelligent navigation positioning module is used for the user to carry out travel navigation, the rehabilitation training module comprises a rehabilitation training video storage module and a user card punching module, and the rehabilitation training module is used for the user to carry out rehabilitation training. The intelligent navigation positioning module comprises a gesture command input module and an electronic map module, a user can input a specific gesture command through the gesture command input module, the specific gesture command corresponds to a specific location query instruction, the specific location query instruction is sent to the electronic map module, and a corresponding navigation route is obtained, for example, a gesture '1' represents a search instruction of a nearby supermarket, a gesture '2' represents a search instruction of a nearby library, and the like.
This system still includes wireless mobile terminal and intelligent pronunciation bracelet, sign language recognition unit intelligent alarm unit time record module the information conversion processing unit medical treatment auxiliary unit with mutual display element 1 all sets up on the wireless mobile terminal, as shown in fig. 2, still be provided with the miniature pickup miaow head 8 that is used for typeeing ordinary communication person's speech information on the intelligent pronunciation bracelet and be used for controlling the control switch 9 that miniature pickup miaow head 8 opened and close, miniature pickup miaow head 8 with control switch 9 with voice text conversion module is connected, wireless mobile terminal with intelligent pronunciation bracelet carries out information exchange through wireless mode.
This system still includes the power supply management unit, the power supply management unit includes that low-voltage reminds module, battery management module, lithium cell, the module includes the battery electric quantity board is reminded to the low-voltage, battery management module includes the PMU chip, the module is reminded to the low-voltage with the lithium cell all with the battery management module electricity is connected.
As shown in fig. 3, sign language information input module 2 includes two mesh cameras and bracing piece, sign language information input module 2 sets up on the wireless mobile terminal, two mesh cameras set up on the bracing piece, the bracing piece card is put at ordinary times on the wireless mobile terminal, will during the use the bracing piece pulls open.
As shown in fig. 4, the sign language information recognition module adopts a sign language recognition algorithm based on deep learning, and the sign language recognition algorithm includes: a. the sign language information input module 2 collects gesture images and videos to construct a static sign language data set and a sign language video set, and stores the static sign language data set and the sign language video set into the sign language data storage database; b. preprocessing and segmenting the depth image in the static sign language data set to obtain a target sign part, and obtaining a target sign part according to a formula K-cMγStretching the image contrast by adopting a power law conversion method to realize image contrast enhancement, wherein c and gamma are normal numbers, and the value ranges of K and M are [0, 1%]Performing morphological processing by adopting closed operation, and removing noise in the image by adopting Gaussian filtering to obtain a clear gesture image segmentation result; c. taking the gesture image as input, and adopting a CNN convolutional neural network to perform static isolated word sign language recognition; d. and extracting the gesture features of the video sequence in the sign language video set by using a CNN convolutional neural network, taking the extracted feature sequence as the input of an LSTM network, identifying the obtained feature sequence through the LSTM network, and outputting a sign language identification result to obtain a gesture class label corresponding to each of N features of the feature vector of the video sequence.
It should be noted that the described embodiments of the invention are only preferred ways of implementing the invention, and that all obvious modifications, which are within the scope of the invention, are all included in the present general inventive concept.
Claims (10)
1. An assistance system for communication of disabled persons, comprising: the medical treatment system comprises a sign language identification unit, an intelligent alarm unit, an event record reminding unit, an information conversion processing unit, a medical treatment auxiliary unit and an interactive display unit;
the sign language recognition unit is used for inputting hand posture information of the deaf-mute and sending a recognition result to the voice playing unit for voice playing, the sign language recognition unit comprises a sign language information input module, a sign language information recognition module and a correction output module, and the sign language information input module, the sign language information recognition module and the correction output module are sequentially connected;
the intelligent alarm unit is used for event alarm and comprises an alarm priority setting module, an alarm information caching processing module, an alarm mode selecting module, an online alarm module and an offline alarm module, wherein the alarm mode comprises an active alarm mode and an automatic alarm mode, the active alarm mode is used for autonomous alarm of a user, the automatic alarm mode alarms according to a set threshold value, and when the alarm information caching processing module detects that a related parameter value exceeds the alarm threshold value, alarm information is sent to the online alarm module and the offline alarm module according to the priority level set by the alarm priority setting module;
the event record reminding unit is used for recording the items required to be completed by the user within a period of time and carrying out timed reminding, and comprises an information reminding module and a time recording module;
the information conversion processing unit comprises a voice playing module, a voice text conversion module, a text image conversion module, a database storage module, a cloud server and a user information association list, wherein the voice playing module is used for playing sign language recognition results, the voice playing module comprises a TTS voice chip, a loudspeaker, a voice mode selection module and a volume adjustment module, the voice playing module is connected with the correction output module, the voice text conversion module, the text image conversion module and the interactive display unit are sequentially connected, and the database storage module comprises a sign language data storage database, a user information storage database, an event storage database and an alarm record storage database;
the medical auxiliary unit comprises a user identity verification module, a health evaluation module, an intelligent navigation positioning module and a rehabilitation training module, wherein the user identity verification module is used for verifying whether the identity of a user is legal or not, the health evaluation module is used for storing a case and a rechecking report of the user and outputting the health report of the user, the intelligent navigation positioning module is used for the user to conduct travel navigation, the rehabilitation training module comprises a rehabilitation training video storage module and a user card punching module, and the rehabilitation training module is used for the user to conduct rehabilitation training.
2. The assistance system as claimed in claim 1, wherein the interactive display unit comprises a liquid crystal interactive display, and the interactive display unit is used for displaying text information corresponding to the speech expression content of the communication person, sign language information and projection related video information.
3. The assistance system as claimed in claim 2, wherein the information conversion processing unit further comprises a sign language action 3D reconstruction module, and the 3D reconstruction module is configured to perform 3D reconstruction on the sign language image converted by the text image conversion module and restore a dynamic 3D sign language video through the interactive display unit.
4. The system as claimed in claim 3, further comprising a wireless mobile terminal and an intelligent voice bracelet, wherein the interactive display units are disposed on the wireless mobile terminal, the intelligent voice bracelet is further provided with a micro microphone for inputting voice information of a communication person and a control switch for controlling the micro microphone to be turned on and off, and the micro microphone is connected with the control switch and the voice-text conversion module.
5. The assistance system for the communication of the disabled as claimed in claim 1, further comprising a power management unit, wherein the power management unit comprises a low voltage reminding module, a battery management module and a lithium battery, the low voltage reminding module comprises a battery power board, the battery management module comprises a PMU chip, and the low voltage reminding module and the lithium battery are electrically connected to the battery management module.
6. The assistant system for the communication of disabled people as claimed in claim 1, wherein when the intelligent alarm unit receives the alarm information, the intelligent alarm unit first buffers the alarm information to the alarm information buffering processing module, when another alarm information is received again in a section, the intelligent alarm unit sequentially sends the alarm information according to the default priority of the alarm priority setting module, otherwise, the intelligent alarm unit directly sends the alarm information to the on-line alarm module and the off-line alarm module, the on-line alarm module comprises a GSM auto-dialing module, and the off-line alarm module comprises an audible and visual alarm circuit.
7. The assistance system for the communication of disabled people as claimed in claim 4, wherein the sign language information input module comprises a binocular camera and a support rod, the sign language information input module is arranged on the wireless mobile terminal, the binocular camera is arranged on the support rod, the support rod is usually clamped on the wireless mobile terminal, and the support rod is pulled away when in use.
8. The system as claimed in claim 1, wherein the sign language information recognition module employs a deep learning based sign language recognition algorithm, the sign language recognition algorithm comprising: the sign language information input module collects gesture images and videos to construct a static sign language data set and sign languagesA video set, and storing the static sign language data set and the sign language video set into the sign language data storage database; preprocessing and segmenting the depth image in the static sign language data set to obtain a target sign part, and obtaining a target sign part according to a formula K-cMγStretching the image contrast by adopting a power law conversion method to realize image contrast enhancement, wherein c and gamma are normal numbers, and the value ranges of K and M are [0, 1%]Performing morphological processing by adopting closed operation, and removing noise in the image by adopting Gaussian filtering to obtain a clear gesture image segmentation result; taking the gesture image as input, and adopting a CNN convolutional neural network to perform static isolated word sign language recognition; and extracting the gesture features of the video sequence in the sign language video set by using a CNN convolutional neural network, taking the extracted feature sequence as the input of an LSTM network, identifying the obtained feature sequence through the LSTM network, and outputting a sign language identification result to obtain a gesture class label corresponding to each of N features of the feature vector of the video sequence.
9. The auxiliary system for the communication of the disabled as claimed in claim 4, wherein the information reminding module comprises a vibration module, a fragrance volatilization module and a light module, the vibration module, the fragrance volatilization module and the light module are all arranged in the intelligent voice bracelet, and the fragrance volatilization module comprises a heating copper sheet and a solid fragrance.
10. The assistance system for the communication of disabled people as claimed in claim 1, wherein the intelligent navigation positioning module comprises a gesture command entry module and an electronic map module, a user can enter a specific gesture command through the gesture command entry module, the specific gesture command corresponds to a specific location query instruction, and the specific location query instruction is sent to the electronic map module to obtain a corresponding navigation route.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010910889.2A CN112183216A (en) | 2020-09-02 | 2020-09-02 | Auxiliary system for communication of disabled people |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010910889.2A CN112183216A (en) | 2020-09-02 | 2020-09-02 | Auxiliary system for communication of disabled people |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112183216A true CN112183216A (en) | 2021-01-05 |
Family
ID=73925575
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010910889.2A Pending CN112183216A (en) | 2020-09-02 | 2020-09-02 | Auxiliary system for communication of disabled people |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112183216A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115019385A (en) * | 2021-05-09 | 2022-09-06 | 北京福乐云数据科技有限公司 | Artificial intelligence medical system through sign language exchange |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101614549A (en) * | 2009-07-24 | 2009-12-30 | 深圳市凯立德计算机系统技术有限公司 | Gesture command input method of navigation system and device |
| CN104573621A (en) * | 2014-09-30 | 2015-04-29 | 李文生 | Dynamic gesture learning and identifying method based on Chebyshev neural network |
| CN108470495A (en) * | 2018-04-04 | 2018-08-31 | 温州大学 | A kind of deaf and dumb patient's ac equipment |
| CN108615009A (en) * | 2018-04-24 | 2018-10-02 | 山东师范大学 | A kind of sign language interpreter AC system based on dynamic hand gesture recognition |
| DE102017214012A1 (en) * | 2017-08-10 | 2019-02-14 | Volkswagen Aktiengesellschaft | Method and device for operating a navigation system of a motor vehicle |
| CN109919127A (en) * | 2019-03-20 | 2019-06-21 | 邱洵 | A kind of sign language languages switching system |
| CN110096991A (en) * | 2019-04-25 | 2019-08-06 | 西安工业大学 | A kind of sign Language Recognition Method based on convolutional neural networks |
| CN110717344A (en) * | 2019-10-12 | 2020-01-21 | 深圳市热丽泰和生命科技有限公司 | Auxiliary communication system based on intelligent wearable equipment |
| US20200075011A1 (en) * | 2018-08-31 | 2020-03-05 | Baidu Online Network Technology (Beijing) Co., Ltd. | Sign Language Information Processing Method and Apparatus, Electronic Device and Readable Storage Medium |
| WO2020119496A1 (en) * | 2018-12-14 | 2020-06-18 | 深圳壹账通智能科技有限公司 | Communication method, device and equipment based on artificial intelligence and readable storage medium |
-
2020
- 2020-09-02 CN CN202010910889.2A patent/CN112183216A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101614549A (en) * | 2009-07-24 | 2009-12-30 | 深圳市凯立德计算机系统技术有限公司 | Gesture command input method of navigation system and device |
| CN104573621A (en) * | 2014-09-30 | 2015-04-29 | 李文生 | Dynamic gesture learning and identifying method based on Chebyshev neural network |
| DE102017214012A1 (en) * | 2017-08-10 | 2019-02-14 | Volkswagen Aktiengesellschaft | Method and device for operating a navigation system of a motor vehicle |
| CN108470495A (en) * | 2018-04-04 | 2018-08-31 | 温州大学 | A kind of deaf and dumb patient's ac equipment |
| CN108615009A (en) * | 2018-04-24 | 2018-10-02 | 山东师范大学 | A kind of sign language interpreter AC system based on dynamic hand gesture recognition |
| US20200075011A1 (en) * | 2018-08-31 | 2020-03-05 | Baidu Online Network Technology (Beijing) Co., Ltd. | Sign Language Information Processing Method and Apparatus, Electronic Device and Readable Storage Medium |
| WO2020119496A1 (en) * | 2018-12-14 | 2020-06-18 | 深圳壹账通智能科技有限公司 | Communication method, device and equipment based on artificial intelligence and readable storage medium |
| CN109919127A (en) * | 2019-03-20 | 2019-06-21 | 邱洵 | A kind of sign language languages switching system |
| CN110096991A (en) * | 2019-04-25 | 2019-08-06 | 西安工业大学 | A kind of sign Language Recognition Method based on convolutional neural networks |
| CN110717344A (en) * | 2019-10-12 | 2020-01-21 | 深圳市热丽泰和生命科技有限公司 | Auxiliary communication system based on intelligent wearable equipment |
Non-Patent Citations (1)
| Title |
|---|
| 俞显刚: "《基于神经网络的视频手语识别》", 《信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115019385A (en) * | 2021-05-09 | 2022-09-06 | 北京福乐云数据科技有限公司 | Artificial intelligence medical system through sign language exchange |
| CN115019385B (en) * | 2021-05-09 | 2023-01-06 | 北京福乐云数据科技有限公司 | Artificial intelligence medical system through sign language communication |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459290B (en) | Interactive intention determining method and device, computer equipment and storage medium | |
| CN106997243B (en) | Speech scene monitoring method and device based on intelligent robot | |
| CN110598576A (en) | Sign language interaction method and device and computer medium | |
| KR20140042994A (en) | Machine learning based of artificial intelligence conversation system using personal profiling information to be extracted automatically from the conversation contents with the virtual agent | |
| CN113835522A (en) | Sign language video generation, translation and customer service method, device and readable medium | |
| CN114926837B (en) | Emotion recognition method based on human-object space-time interaction behavior | |
| CN111353366A (en) | Emotion detection method and device and electronic equipment | |
| KR101584685B1 (en) | A memory aid method using audio-visual data | |
| CN108510988A (en) | A kind of speech recognition system and method for deaf-mute | |
| CN114724224A (en) | Multi-mode emotion recognition method for medical care robot | |
| CN112183216A (en) | Auxiliary system for communication of disabled people | |
| Prabha et al. | Vivoice-Reading Assistant for the Blind using OCR and TTS | |
| Balcı | Overview of intelligent personal assistants | |
| CN113822187A (en) | Sign language translation, customer service, communication method, device and readable medium | |
| Seebun et al. | Let’s talk: An assistive mobile technology for hearing and speech impaired persons | |
| CN113851029A (en) | Barrier-free communication method and device | |
| CN113689660A (en) | Safety early warning method of wearable device and wearable device | |
| Sansen et al. | The Roberta IRONSIDE project a cognitive and physical robot coach for dependent persons | |
| CN115271002B (en) | Identification method, first-aid decision method, medium and life health intelligent monitoring system | |
| CN115937853A (en) | Document generation method, document generation device, electronic device, and storage medium | |
| Joshi et al. | Sign Language Certification Platform with Action Recognition using LSTM Neural Networks | |
| Sansen et al. | The roberta ironside project: A dialog capable humanoid personal assistant in a wheelchair for dependent persons | |
| Bhuiyan et al. | An Assistance System for Visually Challenged People Based on Computer Vision and IOT | |
| Mohan et al. | A Novel Approach to Record and Narrate the Summary of Conversation for Alzheimer Patient | |
| Saha et al. | Vision maker: An audio visual and navigation aid for visually impaired person |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |