CN105807925A - Flexible electronic skin based lip language identification system and method - Google Patents
Flexible electronic skin based lip language identification system and method Download PDFInfo
- Publication number
- CN105807925A CN105807925A CN201610128437.2A CN201610128437A CN105807925A CN 105807925 A CN105807925 A CN 105807925A CN 201610128437 A CN201610128437 A CN 201610128437A CN 105807925 A CN105807925 A CN 105807925A
- Authority
- CN
- China
- Prior art keywords
- lip
- identification
- unit
- flexible electronic
- electronic skin
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08K—Use of inorganic or non-macromolecular organic substances as compounding ingredients
- C08K3/00—Use of inorganic substances as compounding ingredients
- C08K3/02—Elements
- C08K3/04—Carbon
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08K—Use of inorganic or non-macromolecular organic substances as compounding ingredients
- C08K3/00—Use of inorganic substances as compounding ingredients
- C08K3/02—Elements
- C08K3/08—Metals
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08K—Use of inorganic or non-macromolecular organic substances as compounding ingredients
- C08K7/00—Use of ingredients characterised by shape
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/58—Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/24—Speech recognition using non-acoustical features
- G10L15/25—Speech recognition using non-acoustical features using position of the lips, movement of the lips or face analysis
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/28—Constructional details of speech recognition systems
- G10L15/34—Adaptation of a single recogniser for parallel processing, e.g. by use of multiple processors or cloud computing
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08J—WORKING-UP; GENERAL PROCESSES OF COMPOUNDING; AFTER-TREATMENT NOT COVERED BY SUBCLASSES C08B, C08C, C08F, C08G or C08H
- C08J2383/00—Characterised by the use of macromolecular compounds obtained by reactions forming in the main chain of the macromolecule a linkage containing silicon with or without sulfur, nitrogen, oxygen, or carbon only; Derivatives of such polymers
- C08J2383/04—Polysiloxanes
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08K—Use of inorganic or non-macromolecular organic substances as compounding ingredients
- C08K3/00—Use of inorganic substances as compounding ingredients
- C08K3/02—Elements
- C08K3/08—Metals
- C08K2003/085—Copper
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08K—Use of inorganic or non-macromolecular organic substances as compounding ingredients
- C08K2201/00—Specific properties of additives
- C08K2201/011—Nanostructured additives
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08L—COMPOSITIONS OF MACROMOLECULAR COMPOUNDS
- C08L2203/00—Applications
- C08L2203/16—Applications used for films
-
- C—CHEMISTRY; METALLURGY
- C08—ORGANIC MACROMOLECULAR COMPOUNDS; THEIR PREPARATION OR CHEMICAL WORKING-UP; COMPOSITIONS BASED THEREON
- C08L—COMPOSITIONS OF MACROMOLECULAR COMPOUNDS
- C08L2203/00—Applications
- C08L2203/20—Applications use in electrical or conductive gadgets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
Abstract
The invention relates to a flexible electronic skin based lip language identification system and method. The system comprises a lip language signal acquisition unit, a character identification unit, a character display unit and the like. According to the system and method, a flexible electronic skin sensor is used for detecting the movement at the periphery of the lip; the system is simple in structure, light in weight, high in reliability, convenient to carry, low in cost and favorable for industrialization; a lip language identification result can be output in a voice manner through an intelligent apparatus and can be output through a display screen in real time and broadcast through a loudspeaker; or the identification result is displayed in a terminal such as a mobile phone, a computer or the like and is made into a mobile phone APP, so that the identification result can be displayed in mobile phone software in real time and read in the mobile phone, and higher efficiency and more convenience are achieved; and an apparatus can realize real-time communication between a deaf-mute and a normal person, serves as a novel auxiliary sound production device, and has relatively high practicality.
Description
Technical field
The present invention relates to the fields such as sensor, signal processing technology and radio sensing network, particularly relate to a kind of lip reading identification system and method based on flexible electronic skin, belong to intelligent sound identification field.
Background technology
Intelligence wearable device is as a kind of novel wearable device, and its research temperature constantly promotes, and its application is also more and more extensive.Wearable device is not only a kind of hardware device, realizes powerful function alternately by software support and data interaction, high in the clouds especially, and our life, perception will be brought very big transformation by wearable device.
Exchange considerable hurdle between current deaf mute and normal person, cause deaf mute to live communication inconvenience.Although sign language solves this problem, but limitation is very big.Because major part normal person cannot understand the sign language of deaf mute, and when expressing a meaning equally with sign language and lip reading, the expression effect of sign language is more very different than language, and efficiency is non-normally low.
Present lip reading identification technology, each time chart picture of lip vibration when being all speak with video camera shooting, then by the method for image procossing and the recognizer writing complexity, the characteristic quantity extracting image carries out the identification of lip reading.This method with it is proposed that directly measure speak time the vibration of lip surrounding the signal of telecommunication to compared with identifying the method for lip reading, not only realize difficulty big, complicated operation, complex operation step, be inconvenient to carry, equipment needed thereby is many, and needs to use the contour capital equipment of video camera, is unfavorable for large-scale industrialization.Additionally, due to the restriction of hardware, also making traditional lip reading identification technological applicability poor, application prospect is undesirable.
Summary of the invention
It is an object of the invention to propose a kind of lip reading identification system and method based on flexible electronic skin, identification for lip reading, overcome the obstacle exchanged between deaf mute with normal person, and overcome complexity and trouble, the limitation of sign language exchange and poor efficiency that deaf mute and normal person exchange with sign language.
For reaching this purpose, the present invention by the following technical solutions: a kind of lip reading identification system based on flexible electronic skin, including lip reading signal gathering unit, character recognition unit, character image；
Described lip reading signal gathering unit includes flexible electronic skin, produces strain, and export as electronic signals when being used for the motion gathering lip；
Described character recognition unit is for comparing the signal of lip reading signal gathering unit collection with the data of storage in its data base, it is achieved character recognition；
Described character image is for showing recognition result in real time.
Further, described system also includes voice broadcast unit, and described voice broadcast unit is for carrying out real-time broadcasting by recognition result.
Further, described system also includes feedback reminding unit, and described feedback reminding unit is to realize the result according to this identification by 3 LED to remind operation next time, realizes by algorithm is write recognizer.
Further, described flexible electronic skin prepares by the following method: Cu nano wire-graphene film cuts the strip of 2cm × 0.5cm, two ends elargol is stained with copper cash, embed in polydimethylsiloxane (PDMS) again, it is placed in 2h in 70 DEG C of air dry ovens, makes flexible electronic skin.The preparation method of described Cu nano wire-graphene film is as follows:
(1) in 20-25mL ethylene glycol solution, 20-42mgCu nano wire is added, 48-54mg ascorbic acid is added after being uniformly dispersed, 3-3.8mL graphene oxide is added after being uniformly dispersed, move in hydrothermal reaction kettle, it is placed in 120-160 DEG C of air dry oven and reacts 4-6h, it is cooled to room temperature, just obtains Cu nano wire-Graphene composite aquogel.
(2) Cu prepared nano wire-Graphene composite aquogel is placed in 0.5wt% hydrazine hydrate solution dialysis 16h, further take out to pour in 150mL deionized water and be uniformly dispersed, obtain suspension, then with core defecator sucking filtration, just obtain Cu nano wire-graphene film.
Further, described data base includes English alphabet data base and International Phonetic Symbols data base；Set up by the following method: flexible electronic skin is sticked in lip surrounding, when 26 English alphabets or 48 International Phonetic Symbols are read aloud in collection, the strain that lip motion produces, and it is stored in character recognition unit as electronic signals；Each alphabetical or each phonetic symbol has a characteristic of correspondence waveform；
A kind of lip reading recognition methods, comprises the following steps:
(1) flexible electronic skin is sticked in lip surrounding, the strain produced when gathering lip motion, and it is sent to character recognition unit as electronic signals, each phonetic symbol or letter produce a signal waveform；
(2) character recognition unit utilizes data base, by artificial neural network recognizer, each waveform in the signal of telecommunication is identified, and identifies letter or the phonetic symbol of each corresponding wave band；
(3) by character image, recognition result is carried out result to show.
Further, described recognition result is: the letter of each wave band that identification obtains or phonetic symbol are according to sequencing superposition.
The advantage of present system is in that: the intelligent apparatus that this signal of telecommunication vibrated based on mensuration lip is identified is easy to carry, and equipment is simple, and cost is low, and volume is little, and real-time is good, and is beneficial to industrialization, has good application prospect.The terminal of this identification system is finally made the form of mobile phone A PP by us, so can pass through real-time the seeing the result of identification and read aloud the result of identification of this software very easily, thus improving recognition efficiency.It is lip reading identification equally, we this based on survey the lip vibration intelligent apparatus that is identified of the signal of telecommunication than existing video camera constantly shoot speak time lip photo, easy to carry by the lip reading identification technology of image recognition, equipment is simple, cost is low, and volume is little, and real-time is good, and it is beneficial to industrialization, there is good application prospect.
Accompanying drawing illustrates:
Fig. 1 is the structural representation of a kind of wearable device intelligence mask for lip reading identification that the specific embodiment of the invention provides；
Fig. 2 is the structural representation of apparatus of the present invention chip microcontroller identification；
Fig. 3 is that apparatus of the present invention carry out, by wireless sensing and bluetooth, the structural representation that high in the clouds realizes identifying；
Flexible skin sensor is attached to lip surrounding to survey the simulation design sketch of lip vibration during sounding by Fig. 4；
When Fig. 5 is to read English alphabet " A " and " K ", measure the signal of telecommunication being attached to lip surrounding flexible skin sensor with Keithley 2400 table；
When Fig. 6 is with different 3 times English phrases " hello " of tone liaison and " scienceandtechnology ", measure the signal of telecommunication being attached to lip surrounding flexible skin sensor with Keithley 2400 table；
Fig. 7 implements, in claim 5, to be identified the software interface of letter ' M ' and " J " one to one by artificial neural network recognizer MATLAB software；
Fig. 8 implements in claim 5, by the artificial neural network recognizer software interface with MATLAB software identification letter " ZSTU "；
Fig. 9 implements, in claim 6, to be drawn in together by the signal of telecommunication figure of letter " K ", " U " and " Q " by MATLAB software, it can be seen that the superposition of letter K and U is much the same with letter Q really.
Figure 10 implements, in claim 7, namely to be identified the MATLAB interface of letter " Q " by the superposition of letter " K " and " U " by superposition identification；
Figure 11 is the identification process flow diagram flow chart of artificial neural network recognizer.
Detailed description of the invention:
A kind of lip reading identification broadcast device based on flexible electronic skin of the present invention, including lip reading signal gathering unit, character recognition unit, character image, it is also possible to include voice broadcast unit and feedback reminding unit.
Described lip reading signal gathering unit, critical piece is flexible electronic skin, the manufacture method of flexible electronic skin is: Cu nano wire-graphene film cuts the strip of 2cm × 0.5cm, two ends elargol is stained with copper cash, embed again in PDMS, it is placed in 2h in 70 DEG C of air dry ovens, makes flexible electronic skin.Wherein, the preparation method of Cu nano wire-graphene film is as follows:
(1) in 20-25mL ethylene glycol solution, 20-42mgCu nano wire is added, 48-54mg ascorbic acid is added after being uniformly dispersed, 3-3.8mL graphene oxide is added after being uniformly dispersed, move in hydrothermal reaction kettle, it is placed in 120-160 DEG C of air dry oven and reacts 4-6h, it is cooled to room temperature, just obtains Cu nano wire-Graphene composite aquogel.
(2) Cu prepared nano wire-Graphene composite aquogel is placed in 0.5wt% hydrazine hydrate solution dialysis 16h, further take out to pour in 150mL deionized water and be uniformly dispersed, obtain suspension, then with core defecator sucking filtration, just obtain Cu nano wire-graphene film.
Flexible electronic skin has higher susceptiveness and stability, and its shape and size can be customized according to everyone nozzle type, it is ensured that flexible electronic skin can be close to the surrounding of lip, the vibration of lip when sensitive collection is spoken.
Described character recognition unit, the signal of telecommunication exported by flexible electronic skin, uses the recognizer of artificial neural network that the character of the signal of telecommunication collected Yu foundation being judged, data base contrasts, exports immediate result.
Data base sets up by training, and before deaf mute uses, first has to the nozzle type according to deaf mute and pronunciation custom makes and flexible skin sensor can be made completely to be close to the intelligent apparatus of lip surrounding.Then need to pay deaf mute to be trained, obtain meeting the lip reading vibration signal of telecommunication of 26 letters of user pronunciation custom and 48 International Phonetic Symbols, these signals are set up a data base.
In identification process, deaf mute brings the intelligent apparatus meeting oneself nozzle type that we design, when deaf mute speaks, owing to the change of nozzle type makes lip four weekly assembly produce vibration, it is close to the flexible skin sensor of lip to follow the vibration of lip simultaneously and can produce the Light deformation on surface, thus causing the deformation of foil gauge in resistance strain gage sensor, foil gauge resistance value is changed, so that the magnitude of voltage of externally output also can change on foil gauge, this voltage signal is input to the analog acquisition port of single-chip microcomputer, the single-chip microcomputer analog voltage signal to collecting carries out analog digital conversion, become reflecting the digital quantity of each pronunciation of words of lip reading.Single-chip microcomputer obtains the digital quantity of the reflection each pronunciation of words of lip reading, recognizer by the artificial neural network of lip reading identification, signal in data base and these signals are carried out matching identification or superposition identification one by one, obtain the result of lip reading identification, for the common technology means that are shown as of recognition result, voice driven circuit can be directly connected on the one hand and recognition result is read out by speaker；On the other hand, it is also possible to the pronunciation result identified is integrated and the screening of reasonable algorithm obtains the spelling of letter and demonstrates alphabetical spelling by the small display screen outside intelligent apparatus.
Signal of telecommunication transmission between lip reading signal gathering unit and character recognition unit, for technological means commonly used in the art, it is possible to adopt copper cash as above directly to transmit, it is also possible to be transmitted by wireless network.The program of artificial neural network is write single-chip microcomputer (in character recognition unit), incoming for the signal of telecommunication collected single-chip microcomputer, in the process that single-chip microcomputer interior-excess now identifies, and exports result by serial communication interface.The incoming high in the clouds of data (character recognition unit) that will can also be collected very easily by the blue tooth interface of intelligent apparatus or home control network communication protocol, realize the identification of character beyond the clouds, terminal (character image) is delivered to by counter for recognition result, due to high in the clouds character vector storehouse more comprehensively and sufficient so that discrimination improves.
Described character image, it is possible to be a small-sized display screen, shown by the recognition result of character recognition unit in display screen, for prior art well known in the art.Such as, being connected with the serial line interface of display driver circuit by the serial line interface of character recognition unit, the display driver circuit of display device drives the character of display screen display lip reading identification by data/address bus and address bus.The serial line interface of blue tooth interface circuit can also be connected with the serial line interface of single-chip microcomputer, by bluetooth, lip reading recognition result is shown in the terminal such as mobile phone or computer.
Described voice broadcast unit, it is possible to be a little speaker.The recognition result of character recognition unit is carried out real-time broadcasting by speaker, for prior art well known in the art.Such as, the EBI of single-chip microcomputer is connected with voice driven circuit, is then attached to the port of speaker, by the voice broadcast program of write in single-chip microcomputer by the result of identification by real-time the reading out of speaker.
Intelligent apparatus also includes feedback user and plays 3 LED of reminding effect.When identifying after successfully, middle green LED lamp is bright, is used for reminding user, and this identification is over, it is possible to carry out identification next time；When can not identifying result, then the yellow LED lamp on right side is bright, is used for reminding user that this identification is broken down, can not identify result, again lip reading pronunciation of typing；When, after three same typings, still can not identifying result, the red LED lamp being at this moment positioned at the lower left corner is bright, reminds the identification of this word of user or phrase to have no result, and user needs to be said differently or skip this word or expression.Above-mentioned functions can be realized by simple logic circuit, does not do detailed statement at this.
The recognizer of above-mentioned neutral net is: artificial neural network ANN, is a kind of engineering system simulating its structure and intelligent behavior on the understanding basis to human brain tissue structure and operating mechanism.Neural network filter process is divided into two steps, first it is learning process, by a large amount of learning samples, network is trained, constantly connection weights and threshold value are adjusted according to certain learning rules, finally making network have certain desired output, namely this output be correctly to be categorized into by training sample in its generic, now it is believed that network is study has arrived the inherent law between input sample.Followed by categorizing process, apply weights and threshold value that above learning process trains, the sample of arbitrary feeding network is classified.
Owing to, in English, it is aphonic for having some letter in a lot of word, this results in us when identifying word with the phonetic symbol of pronunciation or syllable, although the sound sent in speaker is accurately, but the spelling of the word of display is wrong, causes discrimination to reduce.Therefore, the word of display finally can once be audited by we when writing identification software, can letter aphonic in English word, and namely so-called mute adds according to the word-building of English pronunciation.On this basis, we will change into more comprehensively English pronunciation rule and identify and the algorithm of screening adds the later stage.
The terminal of this identification system finally can being made the form of mobile phone A PP, as long as so opening this software just can see the result of identification very easily in real time, and reading aloud the result of identification, thus more convenient and in hgher efficiency.
Technical scheme is further illustrated below in conjunction with accompanying drawing and by detailed description of the invention.Should be appreciated that specific embodiment described herein is only in order to explain the present invention, is not intended to limit the present invention.
The present invention provides a kind of lip reading recognition methods.
Embodiment 1
A kind of lip reading identification broadcast device intelligence mask based on Graphene flexible electronic skin, schematic appearance is as shown in Figure 1, as shown in Figures 2 and 3, this intelligence Mask devices includes attachment structure block diagram: lip reading signal gathering unit, character recognition unit, character image, voice broadcast unit, feedback reminding unit.It specifically includes that the flexible skin sensor being positioned within mask, it is positioned at the small display screen outside mask and display drives single channel, it is arranged in single-chip microcomputer and the peripheral circuit thereof of intelligence mask, it is positioned at the speaker outside mask and voice driven circuit, for 3 LED warning lights green, red, yellow of feedback user.Intelligence mask also includes blue tooth interface circuit, the wireless communication interface being connected with the exterior terminal such as mobile phone, computer, gives the lithium battery of every part power supply.
As shown in Figure 4, flexible electronic skin sensor is positioned at the innermost layer of mask, and the making material of flexible skin sensor is the extraordinary new material Graphene of electric conductivity so that its susceptiveness and stability are very good.The surrounding of user's lip it is close to, for gathering the signal of telecommunication of lip vibration when speaking during use.And the shape of flexible electronic skin needs the nozzle type according to user to make to measure, it is ensured that sensor can fully gather the characteristic quantity of lip vibration during user pronunciation.
As shown in Figure 2,3, described character recognition unit, the lip collected vibration is converted into the signal of telecommunication by strain transducer, uses the recognizer of artificial neural network that the character of the signal of telecommunication collected Yu foundation being judged, data base contrasts, export immediate result.The process of this identification has two ways to realize.A kind of is load miniature single-chip microcomputer in intelligence Mask devices, the program of artificial neural network is write in single-chip microcomputer, incoming for the signal of telecommunication collected single-chip microcomputer, in the process that single-chip microcomputer interior-excess now identifies, and exports result by serial communication interface.Another kind is, if having network or bluetooth, the incoming high in the clouds of data that just will be able to be collected very easily by the blue-tooth device of intelligence mask or wireless communication interface, realize the identification of character beyond the clouds, terminal is delivered to by counter for recognition result, due to high in the clouds character vector storehouse more comprehensively and sufficient so that discrimination improves.
Described character image, outer layer at intelligence mask is provided with a small-sized display screen, being connected with the serial line interface of display driver circuit by the serial line interface of single-chip microcomputer, the display driver circuit of display device drives the character of display screen display lip reading identification by data/address bus and address bus.The serial line interface of blue tooth interface circuit can also be connected with the serial line interface of single-chip microcomputer, by bluetooth, lip reading recognition result is shown in the terminal such as mobile phone or computer.
Described voice broadcast unit, intelligence mask be arranged above a little speaker.The EBI of single-chip microcomputer is connected with voice driven circuit, is then attached to the port of speaker, by the voice broadcast program of write in single-chip microcomputer by the result of identification by real-time the reading out of speaker.
3 LED usings method that intelligence mask plays reminding effect to feedback user are: middle green LED lamp is positioned at the middle of intelligence mask, is used for reminding user, and this identification is over, it is possible to carry out identification next time；The yellow LED lamp on right side is positioned at the upper right corner of intelligence mask, for reminding user that this identification is broken down, can not identify result, again lip reading pronunciation of typing, when, after three same typings, still can not identifying result, the red colored lamp being at this moment positioned at the lower left corner can be bright, the identification reminding this word of user or phrase is had no result, and user needs to be said differently or skip this word or expression.
Result, for the wireless communication module communicated with extraneous high in the clouds computing unit, by the storage of the complete paired data of high in the clouds computing unit, calculating, identification, and is sent to mobile terminal device by intelligence mask, and described wireless communication module is connected with signal processing unit.
As shown in Figure 5, when reading English alphabet " A " and " K ", the signal of telecommunication being attached to lip surrounding flexible skin sensor is measured with Keithley 2400 table, can be seen that from oscillogram the characteristic quantity of the oscillogram of each letter is different, there is obvious diversity, so that the identification realizing letter is possibly realized.
As shown in Figure 6, during with different 3 times English phrases " hello " of tone liaison and " scienceandtechnology ", the signal of telecommunication being attached to lip surrounding flexible skin sensor is measured with Keithley 2400 table.From oscillogram it can be seen that repeatability is especially good, although the height of tone can affect the amplitude of oscillogram, but the characteristic quantity of waveform is constant, illustrate to realize identifying by the difference of each letter oscillogram characteristic quantity.
Embodiment 2
As shown in Figure 7,8, native system achieves man-to-man identification, also achieves the continuous identification together of several letter.In the identification of character, native system can pass through the algorithm of artificial neural network and realize simple English 26 alphabetical identifications, discrimination height very.And, native system is capable of once identifying 4,5 letters simultaneously, and ensures that the order of letter is constant, and in the recognition result namely exported, letter type and order are completely the same with what input.Such as, lip vibration signal when reading 4 letter " ZSTU " is measured with flexible skin sensor, this signal waveform is inputted the identification system having built up in MATLAB, system can be passed through the recognizer of artificial neural network and each letter in this signal carries out with 26 the alphabetical data bases having built up contrast identification respectively, eventually exports recognition result " ZSTU " on the interface of MATLAB.
Embodiment 3
Such as Fig. 9, shown in 10, native system achieves the superposition identification of letter.MATALB establishes the data base of the lip reading vibration signal of telecommunication of English alphabet " K " and " U ", then lip reading vibration signal of telecommunication when reading English alphabet " Q " is measured, it is inputted the identification system of MATLAB, the pronunciation [kju :] of letter " Q " can be identified the laminated structure of the phonetic symbol [ju :] being the phonetic symbol [kei] of letter " K " and alphabetical " U " by system respectively through the recognizer of neural network, then obtain, further according to the word-building of phonetic symbol, the lip reading vibration signal of telecommunication that this is letter " Q ", finally can export recognition result on the control panel of MATLAB is letter " Q ".
The lip reading identification system set up can not only realize man-to-man identification, moreover it is possible to realizes the power of superposition identification.Namely due to the ultimate unit that syllable is pronunciation, the pronunciation of any word, it is all be decomposed into syllable one by one to read aloud.Lip vibration signal when someone reads 48 International Phonetic Symbols of English is measured respectively with flexible skin sensor, set up a MATLAB data base, then lip vibration signal when he reads any one English word is measured, input a signal in the MATLAB identification system having built up, system can by the recognizer of artificial neural network, by each section of waveform input signal respectively by comparing with the waveform of 48 International Phonetic Symbols data bases having built up successively smoothly, the corresponding International Phonetic Symbols of signal waveform each section or syllable is identified successively by recognizer, then the International Phonetic Symbols these identified in order form word according to the grammer of English pronunciation phonetic symbol, then the interface of MATLAB will reveal whether the letter that identifies.Thus, identifying letter with the superposition identification of phonetic symbol, the superposition identification followed by letter identifies phrase, finally identifies sentence with the superposition identification of phrase, by that analogy, finally can be achieved with the identification of complete people's works and expressions for everyday use.
Before deaf mute uses, first have to the nozzle type according to deaf mute and pronunciation custom makes the intelligent mask that flexible skin sensor can be made completely to be close to lip surrounding.Then need to pay deaf mute to be trained, obtain meeting the lip reading vibration signal of telecommunication of 48 International Phonetic Symbols of user pronunciation custom, these signals are set up a data base.
During use, deaf mute brings the intelligent mask meeting oneself nozzle type that we design, when deaf mute speaks, owing to the change of nozzle type makes lip four weekly assembly produce vibration, it is close to the flexible skin sensor of lip to follow the vibration of lip simultaneously and can produce the Light deformation on surface, thus causing the deformation of foil gauge in resistance strain gage sensor, foil gauge resistance value is changed, so that the magnitude of voltage of externally output also can change on foil gauge, this voltage signal is input to the analog acquisition port of single-chip microcomputer, the single-chip microcomputer analog voltage signal to collecting carries out analog digital conversion, become reflecting the digital quantity of each pronunciation of words of lip reading.Single-chip microcomputer obtains the digital quantity of the reflection each pronunciation of words of lip reading, recognizer by the artificial neural network of lip reading identification, signal in data base and these signals are carried out matching identification or superposition identification one by one, obtain the result of lip reading identification, for recognition result, a reverse side is directly connected to voice driven circuit and recognition result is read out by speaker；On the other hand, the pronunciation result identified is integrated and the screening of reasonable algorithm obtains the spelling of letter and demonstrates alphabetical spelling by the small display screen outside intelligence mask.
The identification process flow diagram flow chart of artificial neural network recognizer as shown in figure 11, the recognizer of above-mentioned neutral net is: artificial neural network ANN, is a kind of engineering system simulating its structure and intelligent behavior on the understanding basis to human brain tissue structure and operating mechanism.Neural network filter process is divided into two steps, first it is learning process, by a large amount of learning samples, network is trained, constantly connection weights and threshold value are adjusted according to certain learning rules, finally making network have certain desired output, namely this output be correctly to be categorized into by training sample in its generic, now it is believed that network is study has arrived the inherent law between input sample.Then it is exactly categorizing process, applies weights and threshold value that above learning process trains, the sample of arbitrary feeding network is classified.
Owing to, in English, it is aphonic for having some letter in a lot of word, this results in us when identifying word with the phonetic symbol of pronunciation or syllable, although the sound sent in speaker is accurately, but the spelling of the word of display is wrong, causes discrimination to reduce.Therefore, the word of display finally can once be audited by we when writing identification software, can letter aphonic in English word, and namely so-called mute adds according to the word-building of English pronunciation.On this basis, later stage will change into, more comprehensively English pronunciation rule, the algorithm identified and screen and add, such as " vowel rules of pronunciation in stressed syllable ", " rules of pronunciation of vowel combination " etc., improve the accuracy rate of identification with this.
The terminal of this identification system finally can being made the form of mobile phone A PP, as long as so opening this software just can see the result of identification very easily in real time, and reading aloud the result of identification, thus more convenient and in hgher efficiency.
Claims (7)
1. the lip reading identification system based on flexible electronic skin, it is characterised in that: include lip reading signal gathering unit, character recognition unit, character image；
Described lip reading signal gathering unit includes flexible electronic skin, produces strain, and export as electronic signals when being used for the motion gathering lip；
Described character recognition unit is for comparing the signal of lip reading signal gathering unit collection with the data of storage in its data base, it is achieved character recognition；
Described character image is for showing recognition result in real time.
2. system according to claim 1, it is characterised in that described device also includes voice broadcast unit, described voice broadcast unit is for carrying out real-time broadcasting by recognition result.
3. system according to claim 1, it is characterized in that, described device also includes feedback reminding unit, and described feedback reminding unit is to realize the result according to this identification by 3 LED to remind operation next time, realizes by algorithm is write recognizer.
4. system according to claim 1, it is characterized in that, described flexible electronic skin prepares by the following method: Cu nano wire-graphene film cuts the strip of 2cm × 0.5cm, two ends elargol is stained with copper cash, embed in polydimethylsiloxane (PDMS) again, it is placed in 2h in 70 DEG C of air dry ovens, makes flexible electronic skin.The preparation method of described Cu nano wire-graphene film is as follows:
(1) in 20-25mL ethylene glycol solution, 20-42mgCu nano wire is added, 48-54mg ascorbic acid is added after being uniformly dispersed, 3-3.8mL graphene oxide is added after being uniformly dispersed, move in hydrothermal reaction kettle, it is placed in 120-160 DEG C of air dry oven and reacts 4-6h, it is cooled to room temperature, just obtains Cu nano wire-Graphene composite aquogel.
(2) Cu prepared nano wire-Graphene composite aquogel is placed in 0.5wt% hydrazine hydrate solution dialysis 16h, further take out to pour in 150mL deionized water and be uniformly dispersed, obtain suspension, then with core defecator sucking filtration, just obtain Cu nano wire-graphene film.
5. system according to claim 1, it is characterised in that described data base includes English alphabet data base and International Phonetic Symbols data base；Set up by the following method: flexible electronic skin is sticked in lip surrounding, when 26 English alphabets or 48 International Phonetic Symbols are read aloud in collection, the strain that lip motion produces, and it is stored in character recognition unit as electronic signals；Each alphabetical or each phonetic symbol has a characteristic of correspondence waveform；
6. the recognition methods of system described in a claim 1, it is characterised in that comprise the following steps:
(1) flexible electronic skin is sticked in lip surrounding, the strain produced when gathering lip motion, and it is sent to character recognition unit as electronic signals, each phonetic symbol or letter produce a signal waveform；
(2) character recognition unit utilizes data base, by artificial neural network recognizer, each waveform in the signal of telecommunication is identified, and identifies letter or the phonetic symbol of each corresponding wave band；
(3) by character image, recognition result is carried out result to show.
7. method according to claim 6, it is characterised in that described recognition result is: after the letter of each wave band that identification obtains or phonetic symbol are overlapped according to sequencing, the recognition result of composition.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610128437.2A CN105807925A (en) | 2016-03-07 | 2016-03-07 | Flexible electronic skin based lip language identification system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610128437.2A CN105807925A (en) | 2016-03-07 | 2016-03-07 | Flexible electronic skin based lip language identification system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105807925A true CN105807925A (en) | 2016-07-27 |
Family
ID=56467790
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610128437.2A Pending CN105807925A (en) | 2016-03-07 | 2016-03-07 | Flexible electronic skin based lip language identification system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105807925A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106377233A (en) * | 2016-09-09 | 2017-02-08 | 浙江理工大学 | Apex pulsation sensor based on CuNWs-rGO-PDMS composite film of flexible structure |
| CN106408876A (en) * | 2016-09-20 | 2017-02-15 | 浙江理工大学 | Fatigue driving monitoring system and monitoring method based on flexible electronic skin |
| CN106430161A (en) * | 2016-09-09 | 2017-02-22 | 浙江理工大学 | Apical impulse sensor based on reduced graphene oxide film with bilayer buckling structure |
| CN106569599A (en) * | 2016-10-24 | 2017-04-19 | 百度在线网络技术（北京）有限公司 | Method and device used for automatic help seeking |
| CN106618516A (en) * | 2016-09-09 | 2017-05-10 | 浙江理工大学 | Apex beat sensor |
| CN107221341A (en) * | 2017-06-06 | 2017-09-29 | 北京云知声信息技术有限公司 | A kind of tone testing method and device |
| CN107765850A (en) * | 2017-09-22 | 2018-03-06 | 上海交通大学 | A kind of sign Language Recognition based on electronic skin and multi-sensor fusion |
| CN107945625A (en) * | 2017-11-20 | 2018-04-20 | 陕西学前师范学院 | A kind of pronunciation of English test and evaluation system |
| CN108446641A (en) * | 2018-03-22 | 2018-08-24 | 深圳市迪比科电子科技有限公司 | A method of degree of lip-rounding image identification system based on machine learning and passes through face line and identify sounding |
| CN108510988A (en) * | 2018-03-22 | 2018-09-07 | 深圳市迪比科电子科技有限公司 | A kind of speech recognition system and method for deaf-mute |
| CN108831472A (en) * | 2018-06-27 | 2018-11-16 | 中山大学肿瘤防治中心 | A kind of artificial intelligence sonification system and vocal technique based on lip reading identification |
| CN109559751A (en) * | 2019-01-09 | 2019-04-02 | 承德石油高等专科学校 | A kind of shape of the mouth as one speaks conversion mask |
| CN112861791A (en) * | 2021-03-11 | 2021-05-28 | 河北工业大学 | Lip language identification method combining graph neural network and multi-feature fusion |
| CN115294838A (en) * | 2022-02-14 | 2022-11-04 | 浙江理工大学 | Deaf child pronunciation training method and system based on lip language recognition |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000338987A (en) * | 1999-05-28 | 2000-12-08 | Mitsubishi Electric Corp | Utterance start monitor, speaker identification device, voice input system, speaker identification system and communication system |
| CN202352332U (en) * | 2011-11-30 | 2012-07-25 | 李扬德 | Portable type lip language identifier |
| CN103294199A (en) * | 2013-06-09 | 2013-09-11 | 华东理工大学 | Silent information identifying system based on facial muscle sound signals |
| CN104575500A (en) * | 2013-10-24 | 2015-04-29 | 中国科学院苏州纳米技术与纳米仿生研究所 | Application of electronic skin in voice recognition, voice recognition system and voice recognition method |
| CN104801244A (en) * | 2015-04-09 | 2015-07-29 | 浙江理工大学 | Method for preparing three-dimensional graphene-copper nanowire composite aerogel |
-
2016
- 2016-03-07 CN CN201610128437.2A patent/CN105807925A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000338987A (en) * | 1999-05-28 | 2000-12-08 | Mitsubishi Electric Corp | Utterance start monitor, speaker identification device, voice input system, speaker identification system and communication system |
| CN202352332U (en) * | 2011-11-30 | 2012-07-25 | 李扬德 | Portable type lip language identifier |
| CN103294199A (en) * | 2013-06-09 | 2013-09-11 | 华东理工大学 | Silent information identifying system based on facial muscle sound signals |
| CN104575500A (en) * | 2013-10-24 | 2015-04-29 | 中国科学院苏州纳米技术与纳米仿生研究所 | Application of electronic skin in voice recognition, voice recognition system and voice recognition method |
| CN104801244A (en) * | 2015-04-09 | 2015-07-29 | 浙江理工大学 | Method for preparing three-dimensional graphene-copper nanowire composite aerogel |
Non-Patent Citations (1)
| Title |
|---|
| TAKEO YAMADA ET AL: "A Stretchable Carbon Nanotube Strain Sensor for Human-motion Detection", 《NATURE NANOTECHNOLOGY》 * |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106430161B (en) * | 2016-09-09 | 2018-05-22 | 浙江理工大学 | A kind of apex beat sensor of the oxidation graphene film based on double-deck buckling structure |
| CN106618516A (en) * | 2016-09-09 | 2017-05-10 | 浙江理工大学 | Apex beat sensor |
| CN106430161A (en) * | 2016-09-09 | 2017-02-22 | 浙江理工大学 | Apical impulse sensor based on reduced graphene oxide film with bilayer buckling structure |
| CN106377233A (en) * | 2016-09-09 | 2017-02-08 | 浙江理工大学 | Apex pulsation sensor based on CuNWs-rGO-PDMS composite film of flexible structure |
| CN106408876A (en) * | 2016-09-20 | 2017-02-15 | 浙江理工大学 | Fatigue driving monitoring system and monitoring method based on flexible electronic skin |
| CN106569599A (en) * | 2016-10-24 | 2017-04-19 | 百度在线网络技术（北京）有限公司 | Method and device used for automatic help seeking |
| CN106569599B (en) * | 2016-10-24 | 2020-05-01 | 百度在线网络技术（北京）有限公司 | Method and device for automatically seeking help |
| CN107221341A (en) * | 2017-06-06 | 2017-09-29 | 北京云知声信息技术有限公司 | A kind of tone testing method and device |
| CN107765850A (en) * | 2017-09-22 | 2018-03-06 | 上海交通大学 | A kind of sign Language Recognition based on electronic skin and multi-sensor fusion |
| CN107945625A (en) * | 2017-11-20 | 2018-04-20 | 陕西学前师范学院 | A kind of pronunciation of English test and evaluation system |
| CN108446641A (en) * | 2018-03-22 | 2018-08-24 | 深圳市迪比科电子科技有限公司 | A method of degree of lip-rounding image identification system based on machine learning and passes through face line and identify sounding |
| CN108510988A (en) * | 2018-03-22 | 2018-09-07 | 深圳市迪比科电子科技有限公司 | A kind of speech recognition system and method for deaf-mute |
| CN108831472A (en) * | 2018-06-27 | 2018-11-16 | 中山大学肿瘤防治中心 | A kind of artificial intelligence sonification system and vocal technique based on lip reading identification |
| CN109559751A (en) * | 2019-01-09 | 2019-04-02 | 承德石油高等专科学校 | A kind of shape of the mouth as one speaks conversion mask |
| CN112861791A (en) * | 2021-03-11 | 2021-05-28 | 河北工业大学 | Lip language identification method combining graph neural network and multi-feature fusion |
| CN115294838A (en) * | 2022-02-14 | 2022-11-04 | 浙江理工大学 | Deaf child pronunciation training method and system based on lip language recognition |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105807925A (en) | Flexible electronic skin based lip language identification system and method | |
| CN105551327A (en) | Interactive pronunciation correcting system and method based on soft electronic skin | |
| CN108000526A (en) | Dialogue exchange method and system for intelligent robot | |
| CN106898197A (en) | A kind of deaf-mute and the equipment of normal person's two-way exchange | |
| CN205050395U (en) | English sound production exerciser | |
| CN107221330A (en) | Punctuate adding method and device, the device added for punctuate | |
| CN105244042B (en) | A kind of speech emotional interactive device and method based on finite-state automata | |
| CN105807924A (en) | Flexible electronic skin based interactive intelligent translation system and method | |
| CN207216921U (en) | A kind of deaf-mute and the Wearable of normal person's two-way exchange | |
| CN107274736A (en) | A kind of interactive Oral English Practice speech sound teaching apparatus in campus | |
| CN101494816A (en) | Hearing-aid device and method suitable for anacusia patient | |
| CN108510988A (en) | A kind of speech recognition system and method for deaf-mute | |
| CN112232127A (en) | Intelligent speech training system and method | |
| CN110503952A (en) | A kind of method of speech processing, device and electronic equipment | |
| CN104361787A (en) | System and method for converting signals | |
| CN206210144U (en) | Gesture language-voice converts cap | |
| CN1331080C (en) | Virtual keyboard and robot control system by brain electric signal | |
| CN115205917A (en) | Man-machine interaction method and electronic equipment | |
| CN110413106B (en) | Augmented reality input method and system based on voice and gestures | |
| CN206907294U (en) | A kind of deaf-mute's Special alternating-current glasses | |
| CN104503572A (en) | Voice-text interaction and conversion device | |
| CN115019820A (en) | Touch sensing and finger combined sounding deaf-mute communication method and system | |
| CN207654512U (en) | Intelligent interaction catharsis instrument | |
| CN214955998U (en) | Voice interaction equipment based on deep learning | |
| CN211427531U (en) | Interactive learning robot device of intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160727 |