CN113450824B - 一种基于多尺度视频特征融合的语音唇读方法及系统 - Google Patents
一种基于多尺度视频特征融合的语音唇读方法及系统 Download PDFInfo
- Publication number
- CN113450824B CN113450824B CN202110716761.7A CN202110716761A CN113450824B CN 113450824 B CN113450824 B CN 113450824B CN 202110716761 A CN202110716761 A CN 202110716761A CN 113450824 B CN113450824 B CN 113450824B
- Authority
- CN
- China
- Prior art keywords
- voice
- video
- lip reading
- module
- scale
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 76
- 238000000034 method Methods 0.000 title claims abstract description 40
- 239000013598 vector Substances 0.000 claims abstract description 67
- 238000001228 spectrum Methods 0.000 claims description 51
- 238000012549 training Methods 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 17
- 238000000605 extraction Methods 0.000 claims description 7
- 238000000926 separation method Methods 0.000 claims description 7
- 238000005520 cutting process Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 3
- 150000001875 compounds Chemical class 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 abstract description 6
- 230000000694 effects Effects 0.000 abstract description 6
- 238000003786 synthesis reaction Methods 0.000 abstract description 6
- 230000002194 synthesizing effect Effects 0.000 abstract description 5
- 230000002123 temporal effect Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 238000009826 distribution Methods 0.000 description 6
- 238000013507 mapping Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000007499 fusion processing Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000000638 solvent extraction Methods 0.000 description 4
- 238000005065 mining Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/24—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being the cepstrum
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
- G10L19/16—Vocoder architecture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Human Computer Interaction (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Multimedia (AREA)
- Acoustics & Sound (AREA)
- Biophysics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Signal Processing (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Studio Circuits (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (6)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110716761.7A CN113450824B (zh) | 2021-06-28 | 2021-06-28 | 一种基于多尺度视频特征融合的语音唇读方法及系统 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110716761.7A CN113450824B (zh) | 2021-06-28 | 2021-06-28 | 一种基于多尺度视频特征融合的语音唇读方法及系统 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113450824A CN113450824A (zh) | 2021-09-28 |
CN113450824B true CN113450824B (zh) | 2022-08-16 |
Family
ID=77813343
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110716761.7A Active CN113450824B (zh) | 2021-06-28 | 2021-06-28 | 一种基于多尺度视频特征融合的语音唇读方法及系统 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113450824B (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114581812B (zh) * | 2022-01-12 | 2023-03-21 | 北京云辰信通科技有限公司 | 视觉语言识别方法、装置、电子设备及存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108664877A (zh) * | 2018-03-09 | 2018-10-16 | 北京理工大学 | 一种基于三维深度数据的动态手势识别方法 |
CN109524006A (zh) * | 2018-10-17 | 2019-03-26 | 天津大学 | 一种基于深度学习的汉语普通话唇语识别方法 |
CN110765868A (zh) * | 2019-09-18 | 2020-02-07 | 平安科技(深圳)有限公司 | 唇读模型的生成方法、装置、设备及存储介质 |
CN112330713A (zh) * | 2020-11-26 | 2021-02-05 | 南京工程学院 | 基于唇语识别的重度听障患者言语理解度的改进方法 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101092820B1 (ko) * | 2009-09-22 | 2011-12-12 | 현대자동차주식회사 | 립리딩과 음성 인식 통합 멀티모달 인터페이스 시스템 |
-
2021
- 2021-06-28 CN CN202110716761.7A patent/CN113450824B/zh active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108664877A (zh) * | 2018-03-09 | 2018-10-16 | 北京理工大学 | 一种基于三维深度数据的动态手势识别方法 |
CN109524006A (zh) * | 2018-10-17 | 2019-03-26 | 天津大学 | 一种基于深度学习的汉语普通话唇语识别方法 |
CN110765868A (zh) * | 2019-09-18 | 2020-02-07 | 平安科技(深圳)有限公司 | 唇读模型的生成方法、装置、设备及存储介质 |
CN112330713A (zh) * | 2020-11-26 | 2021-02-05 | 南京工程学院 | 基于唇语识别的重度听障患者言语理解度的改进方法 |
Non-Patent Citations (1)
Title |
---|
基于发音轨迹的可视语音合成;郑红娜等;《计算机应用与软件》;20130615(第06期);全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN113450824A (zh) | 2021-09-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113192161B (zh) | 一种虚拟人形象视频生成方法、系统、装置及存储介质 | |
CN109308731B (zh) | 级联卷积lstm的语音驱动唇形同步人脸视频合成算法 | |
CN113194348B (zh) | 一种虚拟人讲课视频生成方法、系统、装置及存储介质 | |
US11276231B2 (en) | Semantic deep face models | |
CN112562722A (zh) | 基于语义的音频驱动数字人生成方法及系统 | |
CN112151030B (zh) | 一种基于多模态的复杂场景语音识别方法和装置 | |
CN113378697A (zh) | 一种基于卷积神经网络的说话人脸视频生成方法及装置 | |
CN110853670B (zh) | 音乐驱动的舞蹈生成方法 | |
CN113901894A (zh) | 一种视频生成方法、装置、服务器及存储介质 | |
CN113838174B (zh) | 一种音频驱动人脸动画生成方法、装置、设备与介质 | |
CN111666831A (zh) | 一种基于解耦表示学习的说话人脸视频生成方法 | |
CN112381040B (zh) | 一种基于语音和人脸图像的跨模态生成方法 | |
CN117237521A (zh) | 语音驱动人脸生成模型构建方法、目标人说话视频生成方法 | |
CN114202604A (zh) | 一种语音驱动目标人视频生成方法、装置及存储介质 | |
CN111259785A (zh) | 基于时间偏移残差网络的唇语识别方法 | |
CN113450824B (zh) | 一种基于多尺度视频特征融合的语音唇读方法及系统 | |
Kumar et al. | Robust one shot audio to video generation | |
CN116597857A (zh) | 一种语音驱动图像的方法、系统、装置及存储介质 | |
CN111653270A (zh) | 语音处理方法、装置、计算机可读存储介质及电子设备 | |
CN116233567A (zh) | 一种基于音频情感感知的说话人脸视频生成方法及系统 | |
CN115376482A (zh) | 面部动作视频生成方法及装置、可读介质和电子设备 | |
CN111275778A (zh) | 人脸简笔画生成方法及装置 | |
CN117409121A (zh) | 基于音频和单幅图像驱动的细粒度情感控制说话人脸视频生成方法、系统、设备及介质 | |
CN114360491A (zh) | 语音合成方法、装置、电子设备及计算机可读存储介质 | |
Chatziagapi et al. | AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20231123 Address after: 401135 No. 618 Liangjiang Avenue, Longxing Town, Yubei District, Chongqing Patentee after: Chongqing Research Institute of Wuhan University of Technology Address before: 430070 Hubei Province, Wuhan city Hongshan District Luoshi Road No. 122 Patentee before: WUHAN University OF TECHNOLOGY |
|
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20240511 Address after: No. 19 Zhuoyue Road, Longxing Town, Yubei District, Chongqing, 404100 (cluster registration) Patentee after: Chongqing Wurong Technology Co.,Ltd. Country or region after: China Address before: 401135 No. 618 Liangjiang Avenue, Longxing Town, Yubei District, Chongqing Patentee before: Chongqing Research Institute of Wuhan University of Technology Country or region before: China |
|
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20240708 Address after: 401135 No. 19 Zhuoyue Road, Longxing Town, Liangjiang New Area, Yubei District, Chongqing (cluster registration) Patentee after: Yuezhi Tianxia Digital Technology (Chongqing) Co.,Ltd. Country or region after: China Address before: No. 19 Zhuoyue Road, Longxing Town, Yubei District, Chongqing, 404100 (cluster registration) Patentee before: Chongqing Wurong Technology Co.,Ltd. Country or region before: China |