Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T13/00—Animation
  • G06T13/20—Three-dimensional [3D] animation
  • G06T13/205—Three-dimensional [3D] animation driven by audio data
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T13/00—Animation
  • G06T13/20—Three-dimensional [3D] animation
  • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
  • G10L21/10—Transforming into visible information
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
  • G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
  • G10L21/10—Transforming into visible information
  • G10L2021/105—Synthesis of the lips movements from speech, e.g. for talking heads

Definitions

• avatars animated only by audio signals may have visual movement animations ranging from the very simple, such as mouth flapping with no other motion, through to full motion.
• the mapping of sound to movements may be artificial (that is, unrelated to the sound except that the sound is present or absent), or based on a generic model (that is, using some known sounds to animate some known movements, such as rounding the mouth during an "O" sound), or some combination thereof.
• Current audio-driven animations focus on mouth or mouth and facial expression. Other head and or body motion tends to be absent or wholly artificial and generic to all users of a service.
• Figure 8 shows an example computing device
• an avatar may be considered to comprise a visual digital representation of a user.
• the avatar may change pose during communication with other users.
• the pose changes may be made dependent on audio data received from the user being represented by the avatar.
• the pose changes may comprise changes in head position of the avatar.
• an avatar may be based on a user’s appearance.
• the avatar may be created based on image data corresponding to the user.
• the image data may be used to create a mesh corresponding to a user’s appearance and then image data may be used to overlay further details (e.g., skin, hair, makeup) onto the mesh to create an avatar corresponding to the user’s appearance.
• the avatar may therefore represent a close likeness of the user.
• the user may choose to customize or change the appearance of the avatar such that the avatar does not represent a close likeness of the user.
• An avatar may be considered to comprise a virtual representation of a user.
• the avatar may comprise a representation of the user from the shoulders up or from the neck up.
• mesh 212 may be proprietary geometry designed by a designer (e.g., an artist, etc.) and informed by three dimensional scans of human heads. However, any mesh representation of a head geometry can be used.
• Figure 2A describes an example method 200 of training a base model.
• the example method that can be used to determine a a ‘base’ model f 0 that is considered ‘personalizable’ for a particular user.
• Base model f 0 is trained (or meta-trained) on a pre-existing training dataset.
• the pre-existing dataset will have many users, each with one or more corresponding videos of the user talking.
• the one or more corresponding videos for each user will comprise a small number of videos for each user.
• the small number of videos may be between 1 and 5.
• Each of the videos of the training dataset will have head vertices labelled.
• Each of the videos may comprise audio data synchronized with image data.
• User audio and image data 208 may be captured by image receiving equipment and audio receiving equipment of a user device e.g., of user device 102a. User audio and image data may also be downloaded or uploaded to user device 102a.
• user device 102a receives video data. This may be captured by user device 102a or may be uploaded or downloaded to user device 102a.
• the video data may comprise one or more videos.
• the videos may comprise audio data and synchronized image data.
• the videos may comprise videos of the user operating user device 102a.
• FIG. 6 show an example neural network 600 that could be used in examples for any of neural networks g s , g' e , f' e .
• the neural network is shown as an example only, and neural networks having different structures can be used in methods and systems as described herein. It should also be noted that neural networks having more layers and/or nodes may be used.
• Each node e.g., node 662a, node 662b, node 662c, node 664
• Each edge e.g., edge 666a, edge 666b, edge 666c, etc.
• to compute the value of a node in a given layer take each node that is connected to it from the previous layer and multiply it by its associated edge value/weight, then add these together.
• a nonlinear function e.g., a sigmoid, or tanh
• This is repeated to get the value of each node in a given layer and is repeated for each layer until the output layer is reached.
• the output is a 3-dimensional vector, which could represent the probability the model predicts that the input belongs to 3 possible object classes.
• the neural network has 1 hidden layer of nodes. In the case of a deep neural network which may be used in some examples as a neural network in the methods disclosed herein, there are typically more thanl hidden layers.
• any neural network may be used as an ML model as disclosed herein and Figure 6 is shown as an example only.
• any of the following may be used: sequential variational autoencoders (VAEs); convolutional neural networks; generative models; discriminative models; etc..
• VAEs sequential variational autoencoders
• convolutional neural networks generative models
• discriminative models etc.
• method 900 comprises using the predicted movements of the user to generate animation of an avatar of the user.
• a first aspect there is provided computer-implemented method comprising: receiving, from a user device, video data from a user; training a first machine learning model based on the video data to provide a second machine learning model, the second machine learning model being personalized to the user, wherein the second machine learning model is configured to predict movement of the user based on audio data; receiving further audio data from the user; determining predicted movements of the user based on the further audio data and the second machine learning model; using the predicted movements of the user to generate animation of an avatar of the user.
• At least one of the first machine learning model, the second machine learning model, the third machine learning model and the fourth machine learning model comprises: a convolutional and/or a sequential neural network configured to operate on audio data to predict changes in head pose, expression, and lip movements.
• the instructions executable by a processor of the third aspect may be for performing any of the steps of the examples of the method of the first aspect.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Multimedia (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Signal Processing (AREA)
• Computational Linguistics (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Acoustics & Sound (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Data Mining & Analysis (AREA)
• Quality & Reliability (AREA)
• Processing Or Creating Images (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=80119033

Family Applications (1)

Country Status (6)

Families Citing this family (6)

Citations (7)

Family Cites Families (3)

• 2022
  • 2022-01-31 EP EP22154373.9A patent/EP4220565A1/en not_active Withdrawn
• 2023
  • 2023-01-06 KR KR1020247025950A patent/KR20240142448A/ko active Pending
  • 2023-01-06 EP EP23704845.9A patent/EP4473490A1/en active Pending
  • 2023-01-06 CN CN202380019541.6A patent/CN118648026A/zh active Pending
  • 2023-01-06 JP JP2024536188A patent/JP2025505340A/ja active Pending
  • 2023-01-06 US US18/726,789 patent/US20250069308A1/en active Pending
  • 2023-01-06 WO PCT/US2023/010261 patent/WO2023146741A1/en not_active Ceased

Patent Citations (7)

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events