CN108509903A - Lip reading translating equipment based on 3D imaging technique and lip reading interpretation method - Google Patents
Lip reading translating equipment based on 3D imaging technique and lip reading interpretation method Download PDFInfo
- Publication number
- CN108509903A CN108509903A CN201810276020.XA CN201810276020A CN108509903A CN 108509903 A CN108509903 A CN 108509903A CN 201810276020 A CN201810276020 A CN 201810276020A CN 108509903 A CN108509903 A CN 108509903A
- Authority
- CN
- China
- Prior art keywords
- lip reading
- video cameras
- characteristic point
- images
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Abstract
The invention discloses a kind of lip reading translating equipment and lip reading interpretation method based on 3D imaging technique, the lip reading translating equipment includes that an at least 3D video cameras, an identification module, a processing module and an output module, the 3D video cameras is used to obtain the 3D images of a mouth；The identification module characteristic point on 3D images for identification, every 3D images combine corresponding with the characteristic point formed by unique characteristics point；The processing module combines matched nozzle type with characteristic point for identification；The output module is used to export the information corresponding to 3D Image Matching nozzle type.The identification to lip reading can be realized the present invention is based on the lip reading translating equipment of 3D imaging technique and lip reading interpretation method, the user of aphasis is facilitated to use, the image for more accurately obtaining lip reading is more clear accurate model to establish, and then accurately can translate lip reading to come.
Description
Technical field
The present invention relates to a kind of lip reading translating equipment and lip reading interpretation method based on 3D imaging technique.
Background technology
Lip reading is to understand others' word by seeing others' action of lip when speaking, and is kind of the skill for being difficult, needs
A large amount of practice, has some person hard of hearing that can be exchanged with other people using this skill.
The identification of lip reading is very difficult.
Invention content
The technical problem to be solved by the present invention is in order to overcome in the prior art lip reading be difficult to the defect identified, one kind is provided
The image of lip reading can be more accurately obtained, is more clear accurate model to establish, and then can accurately turn over lip reading
Translate the lip reading translating equipment come and lip reading interpretation method.
The present invention is to solve above-mentioned technical problem by following technical proposals：
A kind of lip reading translating equipment based on 3D imaging technique, feature are that the lip reading translating equipment includes at least
One 3D video cameras, an identification module, a processing module and an output module,
The 3D video cameras are used to obtain the 3D images of a mouth；
The identification module characteristic point on 3D images for identification, every 3D images and the spy formed by unique characteristics point
Sign point combination is corresponding；
The processing module combines matched nozzle type with characteristic point for identification；
The output module is used to export the information corresponding to 3D Image Matching nozzle type.
Preferably, the camera is 3D cameras, the camera includes that a RF transmitter and an infrared ray connect
Device is received, the RF transmitter is for emitting infrared speckle field, and the infrared receiver is for receiving infrared speckle field
Feedback, the numbers of the 3D video cameras are 2, and the first 3D video cameras shoot the mouth from front, and the 2nd 3D video cameras are from the
The shooting direction angle of the shooting mouth below one 3D video cameras, the first 3D video cameras and the 2nd 3D video cameras is acute angle, institute
It further includes a concatenation module to state lip reading translating equipment,
Characteristic point on the concatenation module 3D images that 2 3D video cameras obtain for identification, and 2 3D images are led to
The mode for crossing the coincidence of same characteristic features point is sutured to generate a 3D models；
The identification module space characteristics point on 3D model construction layers for identification, every 3D models with by self space
The characteristic point combination that characteristic point is formed is corresponding.
Preferably, the characteristic point on the structure sheaf includes lip characteristic point, dental features point, tongue characteristic point and chin
Characteristic point, the relative position relation of the whole characteristic points of characteristic point combination record in the 3 d space.
Preferably, the lip reading translating equipment further includes a logging modle and an adjustment module,
The 3D video cameras are used to chronologically obtain the 3D images of several mouths；
The logging modle is used to record the phonetic corresponding to every 3D Image Matching nozzle type；
The adjustment module is used to choose the corresponding Chinese character of each phonetic according to the context of whole phonetics；
The output module is used to press the sequential export all Chinese characters.
Preferably, the lip reading translating equipment further includes a training module,
The training module is used to do model training using an a database pair target nozzle type, and the database includes several
Meaning is the 3D images of the target nozzle type.
The present invention also provides a kind of lip reading interpretation method realized using lip reading translating equipment, feature is, the lip
Language translating equipment includes an at least 3D video cameras, and the lip reading interpretation method includes：
The 3D video cameras obtain the 3D images of a mouth；
Identify the characteristic point on 3D images, every 3D images combine corresponding with the characteristic point formed by unique characteristics point；
Matched nozzle type is combined in identification with characteristic point；
Export the information corresponding to 3D Image Matching nozzle type.
Preferably, the camera is 3D cameras, the camera includes that a RF transmitter and an infrared ray connect
Device is received, the RF transmitter is for emitting infrared speckle field, and the infrared receiver is for receiving infrared speckle field
Feedback, the numbers of the 3D video cameras are 2, and the first 3D video cameras shoot the mouth from front, and the 2nd 3D video cameras are from the
The shooting direction angle of the shooting mouth below one 3D video cameras, the first 3D video cameras and the 2nd 3D video cameras is acute angle, institute
Stating lip reading interpretation method includes：
It identifies the characteristic point on the 3D images of 2 3D video cameras acquisition, and 2 3D images is overlapped by same characteristic features point
Mode suture to generate a 3D models；
Identify the space characteristics point on 3D model construction layers, every 3D models and the feature formed by self space characteristic point
Point combination is corresponding.
Preferably, the characteristic point on the structure sheaf includes lip characteristic point, dental features point, tongue characteristic point and chin
Characteristic point, the relative position relation of the whole characteristic points of characteristic point combination record in the 3 d space.
Preferably, the lip reading interpretation method includes：
The 3D video cameras chronologically obtain the 3D images of several mouths；
Record the phonetic corresponding to every 3D Image Matching nozzle type；
The corresponding Chinese character of each phonetic is chosen according to the context of whole phonetics；
By the whole Chinese character of the sequential export.
Preferably, the lip reading interpretation method includes：
Model training is done using an a database pair target nozzle type, the database includes that several meanings are the target mouth
The 3D images of type.
On the basis of common knowledge of the art, above-mentioned each optimum condition can be combined arbitrarily to get each preferable reality of the present invention
Example.
The positive effect of the present invention is that：The present invention is based on the translations of the lip reading translating equipment and lip reading of 3D imaging technique
Method can realize the identification to lip reading, and the user of aphasis is facilitated to use, and the image of lip reading more accurately be obtained, to build
It is vertical to be more clear accurate model, and then accurately can translate lip reading to come.
Description of the drawings
Fig. 1 is the flow chart of 1 lip reading interpretation method of the embodiment of the present invention.
Fig. 2 is another flow chart of 1 lip reading interpretation method of the embodiment of the present invention.
Specific implementation mode
It is further illustrated the present invention below by the mode of embodiment, but does not therefore limit the present invention to the reality
It applies among a range.
Embodiment 1
A kind of lip reading translating equipment based on 3D imaging technique of the present embodiment, the lip reading translating equipment are taken the photograph including 1 3D
Camera.
The camera is 3D cameras, and the camera includes a RF transmitter and an infrared receiver, institute
It states RF transmitter and is used to receive the feedback of infrared speckle field for emitting infrared speckle field, the infrared receiver.
The lip reading translating equipment further includes an identification module, a processing module, an output module, a logging modle, one
Adjust module and a training module.
The 3D video cameras are used to obtain the 3D images of a mouth.
The identification module characteristic point on 3D images for identification, every 3D images and the spy formed by unique characteristics point
Sign point combination is corresponding.
The processing module combines matched nozzle type with characteristic point for identification.
The output module is used to export the information corresponding to 3D Image Matching nozzle type.
The 3D images include structure sheaf and pixel layer.
The present invention can distinguish nozzle type by the spatial model of mouth.
So as to obtain the characteristic point combination of a mouth, its characteristic point combination of similar nozzle type exists identification feature point
Rule is simultaneously very much like.
In addition, the lip reading translating equipment of the present embodiment also has automatic calibration function, specifically：
The 3D video cameras are used to chronologically obtain the 3D images of several mouths.
The logging modle is used to record the phonetic corresponding to every 3D Image Matching nozzle type.
The adjustment module is used to choose the corresponding Chinese character of each phonetic according to the context of whole phonetics.
The output module is used to press the sequential export all Chinese characters.
The logging modle can obtain the information of multiple sequential nozzle type, some nozzle type easily identify can be accurate
Its meaning is obtained, some nozzle type correspond to the pronunciation of multiple words, and language can be accurately obtained by the contextual analysis of entire sentence
Sentence.
The training module is used to do model training using an a database pair target nozzle type, and the database includes several
Meaning is the 3D images of the target nozzle type.
The present embodiment can be trained in identification 3D images by inputting a large amount of nozzle type 3D images corresponding to mouth image
Nozzle type accuracy.
Referring to Fig. 1, using the lip reading translating equipment of the present embodiment, the present embodiment also provides a kind of lip reading interpretation method, packet
It includes：
Step 100, the 3D video cameras obtain the 3D images of a mouth；
Characteristic point in step 101, identification 3D images, every 3D images are combined with the characteristic point formed by unique characteristics point
It is corresponding；
Matched nozzle type is combined in step 102, identification with characteristic point；
Information corresponding to step 103, output 3D Image Matching nozzle type.
Further, the lip reading interpretation method of the present embodiment includes：
Step 200, the 3D video cameras chronologically obtain the 3D images of several mouths.
Characteristic point in step 201, the every 3D images of identification, every 3D images and the characteristic point formed by unique characteristics point
It combines corresponding.
Step 202 combines each characteristic point, and matched nozzle type is combined in identification with characteristic point.
Step 203 records phonetic corresponding to every 3D Image Matching nozzle type.
Step 204 chooses the corresponding Chinese character of each phonetic according to the context of whole phonetics.
Step 205 presses the sequential export all Chinese characters.
The lip reading interpretation method of the present embodiment further includes：Model training is done using an a database pair target nozzle type, it is described
Database includes the 3D images that several meanings are the target nozzle type.
The lip reading translating equipment based on 3D imaging technique and lip reading interpretation method of the present embodiment can be realized to lip reading
Identification, facilitates the user of aphasis to use, and more accurately obtains the image of lip reading, and accurate mould is more clear to establish
Type, and then accurately can translate lip reading to come.
Embodiment 2
The present embodiment is substantially the same manner as Example 1, the difference is that only：
The camera is 3D cameras, and the camera includes a RF transmitter and an infrared receiver, institute
It states RF transmitter and is used to receive the feedback of infrared speckle field, institute for emitting infrared speckle field, the infrared receiver
The number for stating 3D video cameras is 2, and the first 3D video cameras shoot the mouth from front, and the 2nd 3D video cameras are imaged from the first 3D
Shoot the mouth below machine, the shooting direction angle of the first 3D video cameras and the 2nd 3D video cameras be 30 degree of first video camera and
Second video camera is at an angle in order to the variation of chin is observed, so as to infer the form of tongue.
The lip reading translating equipment further includes a concatenation module,
Characteristic point on the concatenation module 3D images that 2 3D video cameras obtain for identification, and 2 3D images are led to
The mode for crossing the coincidence of same characteristic features point is sutured to generate a 3D models；
The identification module space characteristics point on 3D model construction layers for identification, every 3D models with by self space
The characteristic point combination that characteristic point is formed is corresponding.
The processing module nozzle type with features described above point combinations matches for identification.
The output module is used to export the information corresponding to 3D Image Matching nozzle type.
The space characteristics point can find out characteristic point (including lip exterior feature, trigonum trace, people medium) from pixel layer, so
Afterwards using the corresponding structure sheaf characteristic point of pixel layer characteristic point as space characteristics point.
The space characteristics point can also identify the features such as protrusion, the recess of structure as space spy directly from structure sheaf
Sign point.
Characteristic point on the structure sheaf includes lip characteristic point, dental features point, tongue characteristic point and chin characteristic point,
The relative position relation of the whole characteristic points of characteristic point combination record in the 3 d space.
The 3D models can also be by identifying at least three peak point, then by 3D images on the structure sheaf of 3D images
Structure sheaf sutured in such a way that identical peak point overlaps, the peak point includes salient point and concave point, the number that peak point overlaps
Amount is at least 3.
Corresponding, the lip reading interpretation method of the present embodiment includes：
It identifies the characteristic point on the 3D images of 2 3D video cameras acquisition, and 2 3D images is overlapped by same characteristic features point
Mode suture to generate a 3D models.
Identify the space characteristics point on 3D model construction layers, every 3D models and the feature formed by self space characteristic point
Point combination is corresponding.
Matched nozzle type is combined in identification with characteristic point.
Export the information corresponding to 3D Image Matching nozzle type.
The lip reading translating equipment of the present embodiment can obtain a complete mouth 3D model, special using the space of 3D models
The combination of sign point can preferably be corresponded to nozzle type, more accurate to make lip reading translate.
Although specific embodiments of the present invention have been described above, it will be appreciated by those of skill in the art that these
It is merely illustrative of, protection scope of the present invention is defined by the appended claims.Those skilled in the art is not carrying on the back
Under the premise of from the principle and substance of the present invention, many changes and modifications may be made, but these are changed
Protection scope of the present invention is each fallen with modification.
Claims (10)
1. a kind of lip reading translating equipment based on 3D imaging technique, which is characterized in that the lip reading translating equipment includes at least one
3D video cameras, an identification module, a processing module and an output module,
The 3D video cameras are used to obtain the 3D images of a mouth；
The identification module characteristic point on 3D images for identification, every 3D images and the characteristic point formed by unique characteristics point
It combines corresponding；
The processing module combines matched nozzle type with characteristic point for identification；
The output module is used to export the information corresponding to 3D Image Matching nozzle type.
2. lip reading translating equipment as described in claim 1, which is characterized in that the camera is 3D cameras, the camera shooting
Head includes a RF transmitter and an infrared receiver, and the RF transmitter is described for emitting infrared speckle field
Infrared receiver is used to receive the feedback of infrared speckle field, and the number of the 3D video cameras is 2, and the first 3D video cameras are from just
Face shoots the mouth, and the 2nd 3D video cameras shoot the mouth, the first 3D video cameras and second below the first 3D video cameras
The shooting direction angle of 3D video cameras is acute angle, and the lip reading translating equipment further includes a concatenation module,
Characteristic point on the concatenation module 3D images that 2 3D video cameras obtain for identification, and 2 3D images are passed through into phase
The mode overlapped with characteristic point is sutured to generate a 3D models；
The identification module space characteristics point on 3D model construction layers for identification, every 3D models with by self space feature
The characteristic point combination that point is formed is corresponding.
3. lip reading translating equipment as claimed in claim 2, which is characterized in that the characteristic point on the structure sheaf includes lip spy
Point, dental features point, tongue characteristic point and chin characteristic point are levied, the whole characteristic points of characteristic point combination record are in the 3 d space
Relative position relation.
4. lip reading translating equipment as described in claim 1, which is characterized in that the lip reading translating equipment further includes a record mould
Block and an adjustment module,
The 3D video cameras are used to chronologically obtain the 3D images of several mouths；
The logging modle is used to record the phonetic corresponding to every 3D Image Matching nozzle type；
The adjustment module is used to choose the corresponding Chinese character of each phonetic according to the context of whole phonetics；
The output module is used to press the sequential export all Chinese characters.
5. lip reading translating equipment as described in claim 1, which is characterized in that the lip reading translating equipment further includes a training mould
Block,
The training module is used to do model training using an a database pair target nozzle type, and the database includes several meanings
For the 3D images of the target nozzle type.
6. a kind of lip reading interpretation method realized using lip reading translating equipment, which is characterized in that the lip reading translating equipment includes
An at least 3D video cameras, the lip reading interpretation method include：
The 3D video cameras obtain the 3D images of a mouth；
Identify the characteristic point on 3D images, every 3D images combine corresponding with the characteristic point formed by unique characteristics point；
Matched nozzle type is combined in identification with characteristic point；
Export the information corresponding to 3D Image Matching nozzle type.
7. lip reading interpretation method as claimed in claim 6, which is characterized in that the camera is 3D cameras, the camera shooting
Head includes a RF transmitter and an infrared receiver, and the RF transmitter is described for emitting infrared speckle field
Infrared receiver is used to receive the feedback of infrared speckle field, and the number of the 3D video cameras is 2, and the first 3D video cameras are from just
Face shoots the mouth, and the 2nd 3D video cameras shoot the mouth, the first 3D video cameras and second below the first 3D video cameras
The shooting direction angle of 3D video cameras is acute angle, and the lip reading interpretation method includes：
Identify the characteristic point on the 3D images of 2 3D video cameras acquisition, and the side that 2 3D images are overlapped by same characteristic features point
Formula is sutured to generate a 3D models；
Identify the space characteristics point on 3D model construction layers, every 3D models and the feature point group formed by self space characteristic point
It closes corresponding.
8. lip reading interpretation method as claimed in claim 7, which is characterized in that the characteristic point on the structure sheaf includes lip spy
Point, dental features point, tongue characteristic point and chin characteristic point are levied, the whole characteristic points of characteristic point combination record are in the 3 d space
Relative position relation.
9. lip reading interpretation method as claimed in claim 6, which is characterized in that the lip reading interpretation method includes：
The 3D video cameras chronologically obtain the 3D images of several mouths；
Record the phonetic corresponding to every 3D Image Matching nozzle type；
The corresponding Chinese character of each phonetic is chosen according to the context of whole phonetics；
By the whole Chinese character of the sequential export.
10. lip reading interpretation method as claimed in claim 6, which is characterized in that the lip reading interpretation method includes：
Model training is done using an a database pair target nozzle type, the database includes that several meanings are the target nozzle type
3D images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810276020.XA CN108509903B (en) | 2018-03-30 | 2018-03-30 | Lip language translation device and lip language translation method based on 3D imaging technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810276020.XA CN108509903B (en) | 2018-03-30 | 2018-03-30 | Lip language translation device and lip language translation method based on 3D imaging technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108509903A true CN108509903A (en) | 2018-09-07 |
| CN108509903B CN108509903B (en) | 2021-04-02 |
Family
ID=63379565
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810276020.XA Active CN108509903B (en) | 2018-03-30 | 2018-03-30 | Lip language translation device and lip language translation method based on 3D imaging technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108509903B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633683A (en) * | 2019-09-19 | 2019-12-31 | 华侨大学 | Chinese sentence-level lip language recognition method combining DenseNet and resBi-LSTM |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104978886A (en) * | 2015-06-29 | 2015-10-14 | 广西瀚特信息产业股份有限公司 | Sign language interpreting system based on motion sensing technology and processing method |
| CN105488524A (en) * | 2015-11-26 | 2016-04-13 | 中山大学 | Wearable device based lip language identification method and system |
| CN106504751A (en) * | 2016-08-01 | 2017-03-15 | 深圳奥比中光科技有限公司 | Self adaptation lip reading exchange method and interactive device |
-
2018
- 2018-03-30 CN CN201810276020.XA patent/CN108509903B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104978886A (en) * | 2015-06-29 | 2015-10-14 | 广西瀚特信息产业股份有限公司 | Sign language interpreting system based on motion sensing technology and processing method |
| CN105488524A (en) * | 2015-11-26 | 2016-04-13 | 中山大学 | Wearable device based lip language identification method and system |
| CN106504751A (en) * | 2016-08-01 | 2017-03-15 | 深圳奥比中光科技有限公司 | Self adaptation lip reading exchange method and interactive device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633683A (en) * | 2019-09-19 | 2019-12-31 | 华侨大学 | Chinese sentence-level lip language recognition method combining DenseNet and resBi-LSTM |
| CN110633683B (en) * | 2019-09-19 | 2022-03-25 | 华侨大学 | Chinese sentence-level lip language recognition method combining DenseNet and resBi-LSTM |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108509903B (en) | 2021-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10956725B2 (en) | Automated sign language translation and communication using multiple input and output modalities | |
| Rogez et al. | Mocap-guided data augmentation for 3d pose estimation in the wild | |
| KR101303877B1 (en) | Method and apparatus for serving prefer color conversion of skin color applying face detection and skin area detection | |
| CN110991266B (en) | Binocular face living body detection method and device | |
| CN110544301A (en) | Three-dimensional human body action reconstruction system, method and action training system | |
| CN111178120B (en) | Pest image detection method based on crop identification cascading technology | |
| US20180357819A1 (en) | Method for generating a set of annotated images | |
| CN110728191A (en) | Sign language translation method, and MR-based sign language-voice interaction method and system | |
| CN108174108A (en) | The method and apparatus and mobile terminal for effect of taking pictures are adjusted in the terminal | |
| CN107609475B (en) | Pedestrian detection false detection extraction method based on light field camera | |
| CN111597938A (en) | Living body detection and model training method and device | |
| US20210366087A1 (en) | Image colorizing method and device | |
| CN110163567A (en) | Classroom roll calling system based on multitask concatenated convolutional neural network | |
| CN110290287A (en) | Multi-cam frame synchornization method | |
| CN115035546B (en) | Three-dimensional human body posture detection method and device and electronic equipment | |
| Tang et al. | Research on 3D human pose estimation using RGBD camera | |
| CN108509903A (en) | Lip reading translating equipment based on 3D imaging technique and lip reading interpretation method | |
| CN114373044A (en) | Method, device, computing equipment and storage medium for generating three-dimensional face model | |
| CN106991376A (en) | With reference to the side face verification method and device and electronic installation of depth information | |
| CN104104911B (en) | Timestamp in panoramic picture generating process is eliminated and remapping method and system | |
| CN112287909A (en) | Double-random in-vivo detection method for randomly generating detection points and interactive elements | |
| CN107491459A (en) | The search method and device of three-dimensional image | |
| CN109886212A (en) | From the method and apparatus of rolling fingerprint synthesis fingerprint on site | |
| CN111161399B (en) | Data processing method and assembly for generating three-dimensional model based on two-dimensional image | |
| JP7446903B2 (en) | Image processing device, image processing method, and image processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230720 Address after: 201703 Room 2134, Floor 2, No. 152 and 153, Lane 3938, Huqingping Road, Qingpu District, Shanghai Patentee after: Shanghai Qingyan Heshi Technology Co.,Ltd. Address before: 201703 No.206, building 1, no.3938 Huqingping Road, Qingpu District, Shanghai Patentee before: UNRE (SHANGHAI) INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |