EP4283577A3 - Text and audio-based real-time face reenactment - Google Patents
Text and audio-based real-time face reenactment Download PDFInfo
- Publication number
- EP4283577A3 EP4283577A3 EP23201194.0A EP23201194A EP4283577A3 EP 4283577 A3 EP4283577 A3 EP 4283577A3 EP 23201194 A EP23201194 A EP 23201194A EP 4283577 A3 EP4283577 A3 EP 4283577A3
- Authority
- EP
- European Patent Office
- Prior art keywords
- sequence
- sets
- key points
- method generates
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001815 facial effect Effects 0.000 abstract 2
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/205—3D [Three Dimensional] animation driven by audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/18—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being spectral information of each sub-band
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Acoustics & Sound (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Geometry (AREA)
- Biomedical Technology (AREA)
- Biodiversity & Conservation Biology (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
- Image Analysis (AREA)
Abstract
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/251,436 US10789453B2 (en) | 2019-01-18 | 2019-01-18 | Face reenactment |
US16/251,472 US11049310B2 (en) | 2019-01-18 | 2019-01-18 | Photorealistic real-time portrait animation |
US16/509,370 US11114086B2 (en) | 2019-01-18 | 2019-07-11 | Text and audio-based real-time face reenactment |
EP20707896.5A EP3912159B1 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
PCT/US2020/014221 WO2020150688A1 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20707896.5A Division-Into EP3912159B1 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
EP20707896.5A Division EP3912159B1 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
Publications (2)
Publication Number | Publication Date |
---|---|
EP4283577A2 EP4283577A2 (en) | 2023-11-29 |
EP4283577A3 true EP4283577A3 (en) | 2024-02-14 |
Family
ID=69724071
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP23201200.5A Pending EP4283578A3 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
EP20707896.5A Active EP3912159B1 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
EP23201194.0A Pending EP4283577A3 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP23201200.5A Pending EP4283578A3 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
EP20707896.5A Active EP3912159B1 (en) | 2019-01-18 | 2020-01-18 | Text and audio-based real-time face reenactment |
Country Status (4)
Country | Link |
---|---|
EP (3) | EP4283578A3 (en) |
KR (1) | KR102509666B1 (en) |
CN (1) | CN113228163A (en) |
WO (1) | WO2020150688A1 (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112383721B (en) * | 2020-11-13 | 2023-04-07 | 北京有竹居网络技术有限公司 | Method, apparatus, device and medium for generating video |
CN112927712B (en) * | 2021-01-25 | 2024-06-04 | 网易(杭州)网络有限公司 | Video generation method and device and electronic equipment |
CN113066497A (en) * | 2021-03-18 | 2021-07-02 | Oppo广东移动通信有限公司 | Data processing method, device, system, electronic equipment and readable storage medium |
CN113132797A (en) * | 2021-04-22 | 2021-07-16 | 北京房江湖科技有限公司 | Video generation method and device, computer-readable storage medium and electronic equipment |
KR102353556B1 (en) | 2021-11-01 | 2022-01-20 | 강민호 | Apparatus for Generating Facial expressions and Poses Reappearance Avatar based in User Face |
CN114093384A (en) * | 2021-11-22 | 2022-02-25 | 上海商汤科技开发有限公司 | Speaking video generation method, device, equipment and storage medium |
KR20230108869A (en) | 2022-01-12 | 2023-07-19 | 주식회사 이너버즈 | A learning method of a virtual person creation model using deep learning |
US12020386B2 (en) | 2022-06-23 | 2024-06-25 | Snap Inc. | Applying pregenerated virtual experiences in new location |
CN115311731B (en) * | 2022-10-10 | 2023-01-31 | 之江实验室 | Expression generation method and device for sign language digital person |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH05153581A (en) * | 1991-12-02 | 1993-06-18 | Seiko Epson Corp | Face picture coding system |
JP4370811B2 (en) * | 2003-05-21 | 2009-11-25 | カシオ計算機株式会社 | Voice display output control device and voice display output control processing program |
CN100343874C (en) * | 2005-07-11 | 2007-10-17 | 北京中星微电子有限公司 | Voice-based colored human face synthesizing method and system, coloring method and apparatus |
US7738997B2 (en) * | 2005-12-19 | 2010-06-15 | Chyi-Yeu Lin | Robotic system for synchronously reproducing facial expression and speech and related method thereof |
-
2020
- 2020-01-18 WO PCT/US2020/014221 patent/WO2020150688A1/en unknown
- 2020-01-18 EP EP23201200.5A patent/EP4283578A3/en active Pending
- 2020-01-18 EP EP20707896.5A patent/EP3912159B1/en active Active
- 2020-01-18 CN CN202080007732.7A patent/CN113228163A/en active Pending
- 2020-01-18 KR KR1020217026007A patent/KR102509666B1/en active IP Right Grant
- 2020-01-18 EP EP23201194.0A patent/EP4283577A3/en active Pending
Non-Patent Citations (5)
Title |
---|
CARL VONDRICK ET AL: "Generating Videos with Scene Dynamics", 8 September 2016 (2016-09-08), XP055432559, Retrieved from the Internet <URL:https://arxiv.org/pdf/1609.02612.pdf> * |
KONSTANTINOS VOUGIOUKAS ET AL: "End-to-End Speech-Driven Realistic Facial Animation with Temporal GANs", IEEE CONFERENCE ON COMPUTER VISIONA ND PATTERN RECOGNITION WORKSHOPS, 1 January 2019 (2019-01-01), pages 37 - 40, XP055690306, Retrieved from the Internet <URL:http://openaccess.thecvf.com/content_CVPRW_2019/papers/Sight%20and%20Sound/Konstantinos_Vougioukas_End-to-End_Speech-Driven_Realistic_Facial_Animation_with_Temporal_GANs_CVPRW_2019_paper.pdf> [retrieved on 20200429] * |
PHILLIP ISOLA ET AL: "Image-to-Image Translation with Conditional Adversarial Networks", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 21 November 2016 (2016-11-21), XP080733474, DOI: 10.1109/CVPR.2017.632 * |
RITHESH KUMAR ET AL: "ObamaNet: Photo-realistic lip-sync from text", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 6 December 2017 (2017-12-06), XP081203368 * |
SUPASORN SUWAJANAKORN ET AL: "Synthesizing Obama", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 36, no. 4, 20 July 2017 (2017-07-20), pages 1 - 13, XP058372869, ISSN: 0730-0301, DOI: 10.1145/3072959.3073640 * |
Also Published As
Publication number | Publication date |
---|---|
EP4283578A3 (en) | 2024-02-14 |
KR20210119441A (en) | 2021-10-05 |
WO2020150688A1 (en) | 2020-07-23 |
EP3912159B1 (en) | 2023-12-20 |
KR102509666B1 (en) | 2023-03-15 |
EP4283578A2 (en) | 2023-11-29 |
EP3912159A1 (en) | 2021-11-24 |
CN113228163A (en) | 2021-08-06 |
EP4283577A2 (en) | 2023-11-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP4283577A3 (en) | Text and audio-based real-time face reenactment | |
EP3511942A3 (en) | Cross-domain image analysis and cross-domain image synthesis using deep image-to-image networks and adversarial networks | |
US10581625B1 (en) | Automatically altering the audio of an object during video conferences | |
US20170243582A1 (en) | Hearing assistance with automated speech transcription | |
MX2021013237A (en) | Customized output to optimize for user preference in a distributed system. | |
EP3499900A3 (en) | Video processing method, apparatus and device | |
EP4224833A3 (en) | Method and apparatus utilizing residual echo estimate information to derive secondary echo reduction parameters | |
WO2020201999A3 (en) | Pupil tracking system and method, and digital display device and digital image rendering system and method using same | |
US8913189B1 (en) | Audio and video processing associated with visual events | |
TW201624467A (en) | Meeting minutes device and method thereof for automatically creating meeting minutes | |
WO2020106592A3 (en) | Controlling device output according to a determined condition of a user | |
MX2023004329A (en) | Audio generator and methods for generating an audio signal and training an audio generator. | |
EP4235648A3 (en) | Language model biasing | |
EP3629240A3 (en) | Generative adversarial networks for local noise removal from an image | |
US20060129400A1 (en) | Method and system for converting text to lip-synchronized speech in real time | |
WO2020150690A3 (en) | Systems and methods for providing personalized videos | |
TW201624468A (en) | Meeting minutes device and method thereof for automatically creating meeting minutes | |
EP2106121A1 (en) | Subtitle generation methods for live programming | |
EP4084004B1 (en) | Compensation for face coverings in captured audio | |
EP3789928A3 (en) | Neural network method and apparatus | |
KR101881862B1 (en) | Apparatus and Method for Generating Interpretation Text of Medical Image | |
WO2018224839A3 (en) | Methods and systems for generating a reaction video | |
KR20180084469A (en) | Apparatus and method for providing voice data | |
EP3968338A4 (en) | Method and system of generating individual dietary recommendations based on analysis of microbiota composition | |
US20210383722A1 (en) | Haptic and visual communication system for the hearing impaired |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AC | Divisional application: reference to earlier application |
Ref document number: 3912159 Country of ref document: EP Kind code of ref document: P |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Free format text: PREVIOUS MAIN CLASS: G06V0040160000 Ipc: G10L0013040000 |
|
PUAL | Search report despatched |
Free format text: ORIGINAL CODE: 0009013 |
|
AK | Designated contracting states |
Kind code of ref document: A3 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 13/08 20130101ALN20240111BHEP Ipc: G06V 40/10 20220101ALI20240111BHEP Ipc: G06V 10/70 20220101ALI20240111BHEP Ipc: G06V 10/44 20220101ALI20240111BHEP Ipc: G10L 13/04 20130101AFI20240111BHEP |