CN111145282B - Avatar composition method, apparatus, electronic device, and storage medium - Google Patents
Avatar composition method, apparatus, electronic device, and storage medium Download PDFInfo
- Publication number
- CN111145282B CN111145282B CN201911274701.3A CN201911274701A CN111145282B CN 111145282 B CN111145282 B CN 111145282B CN 201911274701 A CN201911274701 A CN 201911274701A CN 111145282 B CN111145282 B CN 111145282B
- Authority
- CN
- China
- Prior art keywords
- expression
- frame
- features
- avatar
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 239000000203 mixture Substances 0.000 title claims description 28
- 230000014509 gene expression Effects 0.000 claims abstract description 332
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 115
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 115
- 238000012549 training Methods 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims description 59
- 230000003287 optical effect Effects 0.000 claims description 49
- 230000004927 fusion Effects 0.000 claims description 22
- 230000008921 facial expression Effects 0.000 claims description 12
- 230000015654 memory Effects 0.000 claims description 11
- 230000002996 emotional effect Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000001308 synthesis method Methods 0.000 abstract description 4
- 210000003205 muscle Anatomy 0.000 description 25
- 230000008451 emotion Effects 0.000 description 14
- 210000001097 facial muscle Anatomy 0.000 description 13
- 238000004891 communication Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 9
- 210000004237 neck muscle Anatomy 0.000 description 8
- 239000011159 matrix material Substances 0.000 description 6
- 210000003128 head Anatomy 0.000 description 5
- 210000005252 bulbus oculi Anatomy 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000001419 dependent effect Effects 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000002194 synthesizing effect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 210000000988 bone and bone Anatomy 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 210000004709 eyebrow Anatomy 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000007935 neutral effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 101100134058 Caenorhabditis elegans nth-1 gene Proteins 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000008485 antagonism Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000000994 depressogenic effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000001061 forehead Anatomy 0.000 description 1
- 230000037433 frameshift Effects 0.000 description 1
- 210000004209 hair Anatomy 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 208000003265 stomatitis Diseases 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7834—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using audio features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the invention provides an avatar synthesis method, an avatar synthesis device, electronic equipment and a storage medium, wherein the method comprises the following steps: determining relevant characteristics of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data; inputting the image data and related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with expressions corresponding to voice data; the expression synthesis model is obtained by training based on relevant features of sample voice data corresponding to the sample speaker video and sample image data. The method, the device, the electronic equipment and the storage medium provided by the embodiment of the invention can enable the virtual image expression to be better attached to the voice data, and are more natural and real.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an avatar composition method, apparatus, electronic device, and storage medium.
Background
In recent years, with the continuous progress of computer speech synthesis and video synthesis technologies, various virtual image synthesis technologies based on speech driving have been developed in the industry. The avatar may perform news broadcasting, weather forecast, narration game, provide order service, etc.
In the process of executing the tasks, most of the virtual images only synthesize mouth shapes matched with output voices, the virtual images always keep neutral expressions, or several basic expressions are preset, and corresponding expressions are configured for different voice output contents. When the synthesized virtual image outputs voice, the corresponding expression is often not lifelike and natural, and the user experience is poor.
Disclosure of Invention
The embodiment of the invention provides an avatar synthesis method, an avatar synthesis device, electronic equipment and a storage medium, which are used for solving the problem that the corresponding expression of the existing avatar is not lifelike and natural when outputting voice.
In a first aspect, an embodiment of the present invention provides an avatar composition method, including:
determining relevant characteristics of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data;
Inputting the image data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with expressions corresponding to the voice data;
the expression synthesis model is trained based on a sample speaker video, relevant characteristics of sample voice data corresponding to the sample speaker video and sample image data.
Preferably, the inputting the avatar data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model specifically includes:
inputting image data and related features corresponding to any frame to a feature extraction layer of the expression synthesis model respectively to obtain frame features output by the feature extraction layer;
and inputting the frame characteristics to an expression prediction layer of the expression synthesis model to obtain a virtual expression image of any frame output by the expression prediction layer.
Preferably, the inputting the image data and the related features corresponding to any frame to the feature extraction layer of the expression synthesis model to obtain the frame features output by the feature extraction layer specifically includes:
Inputting the image data and related features corresponding to any frame to a current feature extraction layer of the feature extraction layer to obtain current features output by the current feature extraction layer;
and inputting the virtual expression image of any frame preset before to a frame front feature extraction layer of the feature extraction layer to obtain the frame front features output by the frame front feature extraction layer.
Preferably, the inputting the frame features to the expression prediction layer of the expression synthesis model to obtain a virtual expression map of the any frame output by the expression prediction layer specifically includes:
and the current features and the pre-frame features are fused and then input into the expression prediction layer, so that a virtual expression image of any frame output by the expression prediction layer is obtained.
Preferably, the fusing the current feature and the pre-frame feature and inputting the fused current feature and the pre-frame feature to the expression prediction layer to obtain a virtual expression map of any frame output by the expression prediction layer, which specifically includes:
the current characteristics and the frame front characteristics are fused and then input into a candidate expression prediction layer of the expression prediction layer, so that a candidate expression image output by the candidate expression prediction layer is obtained;
The current features and the frame front features are fused and then input into an optical flow prediction layer of the expression prediction layer, so that optical flow information output by the optical flow prediction layer is obtained;
and inputting the candidate expression images and the optical flow information into a fusion layer in the expression prediction layer to obtain a virtual expression image of any frame output by the fusion layer.
Preferably, the expression synthesis model is obtained based on a sample speaker video, relevant characteristics of sample voice data corresponding to the sample speaker video and sample image data, and a discriminant training, and the expression synthesis model and the discriminant form a generative type countermeasure network.
Preferably, the arbiter comprises an image arbiter and/or a video arbiter;
the image discriminator is used for judging the synthesis authenticity of any frame of virtual expression graph in the virtual image video, and the video discriminator is used for judging the synthesis authenticity of the virtual image video.
Preferably, the relevant features include language-related features, as well as emotional and/or speaker identity features.
Preferably, the avatar data is determined based on the speaker identity.
Preferably, the expression corresponding to the voice data of the avatar configuration in the avatar video includes a facial expression and a neck expression.
In a second aspect, an embodiment of the present invention provides an avatar composition device including:
a relevant feature determining unit for determining relevant features of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data;
the expression synthesis unit is used for inputting the image data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with an expression corresponding to the voice data;
the expression synthesis model is obtained by training relevant features of sample voice data and sample image data corresponding to a sample speaker video.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a bus, where the processor, the communication interface, and the memory are in communication with each other through the bus, and the processor may invoke logic instructions in the memory to perform the steps of the method as provided in the first aspect.
In a fourth aspect, embodiments of the present invention provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method as provided by the first aspect.
According to the method, the device, the electronic equipment and the storage medium for synthesizing the virtual image, which are provided by the embodiment of the invention, the expression synthesis of the virtual image is performed by applying the related characteristics containing rich expression related information, so that the expression of the virtual image can be better attached to voice data, and the virtual image is more natural and real. In addition, in the virtual image video generated by the expression synthesis model, the expression of the virtual image exists in an integral form, compared with a mode of singly modeling each region of the expression in the virtual image, the method can effectively solve the problem of the linkage of the muscles of each region by aiming at the integral modeling of the expression, so that the muscle linkage of each region is more natural and lifelike.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart illustrating an avatar composition method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of an expression synthesis method according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of a feature extraction method according to an embodiment of the present invention;
fig. 4 is a flow chart of an expression prediction method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an expression synthesis model according to an embodiment of the present invention;
fig. 6 is a flowchart illustrating an avatar composition method according to another embodiment of the present invention;
fig. 7 is a schematic structural view of an avatar composition device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the prior art, the synthesis technology of the avatar is mainly classified into the following three types:
the first category, speech driven avatar composition technology: the language information and expression information acquired from the voice are independently applied to the finally synthesized video. In the method, only a plurality of basic expressions are considered, the synthesized virtual image is compared with a crowing, only a plurality of predefined basic expressions can be made, and the problems that the mouth and the lips are not matched with the eyebrows, the throat, the cheeks and the like exist. The above problems are on the one hand because the opening and closing of the mouth shape is determined only according to the pronunciation characteristics of the voice content, and the difference between different people is not considered, and the physiological linkage between the face muscle blocks is not considered, so that the rich emotion cannot be expressed individually. On the other hand, because such a method can only select one or two from several or tens of fixed expressions to be superimposed on the synthesized video, a rich facial expression cannot be synthesized.
Second, the virtual image synthesis technology based on expression migration: and migrating the facial expression, the mouth shape and the rigid motion of the driving person to the virtual image. The video synthesized by the method is more lifelike, but is very dependent on real person performance, and can not be synthesized offline.
Third, based on the technology of synthesizing the virtual image expression by modeling each part of the face separately, the artist is required to design the motion of the whole face according to the physiological and aesthetic expertise, and a video segment is synthesized to edit the state of each part frame by frame, so that not only is strong expertise required, but also time and effort are consumed.
From the anatomical point of view, the face of the person has 42 muscles, so that rich expressions can be generated, and various different moods and emotions can be accurately conveyed. The stretching of these muscles is not independent, but has a strong correlation, for example: the person speaks in calm state, and the muscle of lips and chin stretches, and the same sentence that the person says when the emotion is excited, and the muscle of forehead, cheek muscle also stretches, and the muscle stretching intensity in areas such as lips, chin is obviously bigger than calm. In addition, there are thousands of human expressions, and the existing method only has several or tens of preset expressions, and the expression capability is not fine enough and personalized. Therefore, how to automatically compose an avatar with more lifelike and natural appearance is still a problem to be solved by those skilled in the art.
In this regard, the embodiment of the present invention provides an avatar composition method. Fig. 1 is a flow chart of an avatar composition method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
Step 110, determining relevant characteristics of voice data; the relevant features are used to characterize the features contained in the speech data that relate to the expression of the speaker.
Specifically, the voice data is voice data for performing avatar synthesis, where the avatar may be an avatar, or may be an avatar, an animal, or the like, and the embodiment of the present invention is not limited thereto. The voice data may be voice data of a speaker speaking collected by the radio device, or may be intercepted from voice data obtained through a network or the like, which is not particularly limited in the embodiment of the present invention.
The relevant features are features related to the expression of the speaker, such as language-related features in the voice data, which correspond to different utterances, which require the speaker to mobilize facial muscles to form different mouth shapes, such as emotional features in the voice data, and when the speaker speaks the same content under different emotions, the movements of the facial muscles including the mouth shapes and neck muscles are also different, such as scene features in the voice data, the speaking scene of the speaker may also affect the facial expression of the speaker, such as when speaking in a noisy environment, the speaker may speak aloud, the facial expression may be relatively exaggerated, when speaking in a quiet environment, the speaker may speak aloud, the facial expression may be relatively fine, such as the speaker's identity features in the voice data, the expressions of different speakers when speaking may be different, such as a host who hosts a child program, the expressions when speaking may be tangential, a host who hosts a smile program, and the host when speaking may be exaggerated.
Step 120, inputting the avatar data and related features into the expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with expressions corresponding to the voice data; the expression synthesis model is obtained by training based on the sample speaker video, the relevant characteristics of the sample voice data corresponding to the sample speaker video and the sample image data.
Specifically, the avatar data, that is, the image data for performing avatar composition, may be an avatar of a speaker corresponding to the voice data, or may be an avatar unrelated to the speaker corresponding to the voice, which is not particularly limited in the embodiment of the present invention. The avatar data includes a texture map and an expression mask map, wherein the texture map is an image of the avatar itself, the texture map includes the avatar, and each region of the avatar where the expression is performed, and the expression mask map is an avatar image after masking each region of the avatar where the expression is performed, and may be one expression mask map corresponding to each frame or one expression mask map corresponding to a plurality of frames.
The expression synthesis model is used for analyzing the expression of the avatar based on the relevant characteristics of the avatar data, and combining the expression of the avatar to obtain an avatar video configured with the expression corresponding to the voice data. Before executing step 120, an expression synthesis model may be trained in advance, and specifically, the expression synthesis model may be trained by the following manner: firstly, a large number of sample speaker videos and sample voice data corresponding to the sample speaker videos are collected, and sample image data in the sample speaker videos and relevant features in the sample voice data are extracted. Here, the sample speaker video is a real person speaker video. And then training the initial model based on the relevant characteristics of the sample voice data corresponding to the sample speaker video and the sample image data, thereby obtaining an expression synthesis model.
According to the method provided by the embodiment of the invention, the expression synthesis of the virtual image is performed by applying the related features containing rich expression related information, so that the expression of the virtual image can be better attached to voice data, and the virtual image is more natural and real. In addition, in the virtual image video generated by the expression synthesis model, the expression of the virtual image exists in an integral form, compared with a mode of singly modeling each region of the expression in the virtual image, the method can effectively solve the problem of the linkage of the muscles of each region by aiming at the integral modeling of the expression, so that the muscle linkage of each region is more natural and lifelike.
Based on the above embodiments, the expression synthesis model includes a feature extraction layer and an expression prediction layer. Fig. 2 is a flow chart of an expression synthesis method according to an embodiment of the present invention, as shown in fig. 2, step 120 specifically includes:
step 121, inputting the image data and the related features corresponding to any frame to the feature extraction layer of the expression synthesis model to obtain the frame features output by the feature extraction layer.
Specifically, the speech data may be divided into speech data of a plurality of frames, for which there is a corresponding correlation feature. Also, in the avatar data, the same texture map may correspond to each frame to embody the appearance of an avatar in the avatar video, and different expression mask maps may correspond to different frames to embody the actions of the avatar corresponding to different frames in the avatar video, in particular, the head actions.
In the expression synthesis model, the feature extraction layer is used for extracting frame features of any frame from image data and related features respectively corresponding to the frame. The frame features herein may be the image features of the frame and the expression related features of the frame, and may also include fusion features of the image features and the expression related features of the frame, which is not particularly limited in the embodiment of the present invention.
Step 122, inputting the frame characteristics into the expression prediction layer of the expression synthesis model to obtain the virtual expression map of the frame output by the expression prediction layer.
Specifically, in the expression synthesis model, the expression prediction layer is used for predicting a virtual expression map of any frame based on frame characteristics of the frame. Here, the avatar is an image including an avatar, wherein the avatar is configured with an expression corresponding to the voice data of the frame, and a position, an action, etc. of the avatar are consistent with the avatar data corresponding to the frame. Each frame of virtual expression image forms an avatar video.
According to the method provided by the embodiment of the invention, the frame characteristics of any frame are obtained, the virtual expression image of the frame is obtained based on the frame characteristics, the virtual image video is finally obtained, and the overall naturalness and fidelity of the virtual image video are improved by improving the naturalness and fidelity of the virtual expression image of each frame.
Based on any of the above embodiments, the feature extraction layer includes a current feature extraction layer and a pre-frame feature extraction layer; fig. 3 is a flow chart of a feature extraction method according to an embodiment of the present invention, as shown in fig. 3, step 121 specifically includes:
and 1211, inputting the image data and the related features corresponding to any frame respectively to a current feature extraction layer of the feature extraction layer to obtain the current features output by the current feature extraction layer.
Step 1212, inputting the virtual expression image of the frame pre-set frame to the frame pre-feature extraction layer of the feature extraction layer to obtain the frame pre-feature output by the frame pre-feature extraction layer.
Specifically, the frame characteristics of any frame comprise two parts, namely a current characteristic and a frame front characteristic, wherein the current characteristic is obtained by extracting characteristics of image data and related characteristics respectively corresponding to the frame through a current characteristic extraction layer, and the current characteristic is used for reflecting the characteristics of the frame in the aspect of the virtual image, particularly the expression of the virtual image; the pre-frame features are obtained by extracting features of the virtual expression map of the preset frame before the frame through a pre-frame feature extraction layer, and are used for reflecting the virtual images, especially the features of the virtual image expression, in the virtual expression map of the preset frame before the frame.
Here, any frame pre-set frame may be a number of frames before the frame that are pre-set, for example, any frame is the nth frame, and the frame pre-set frame of the frame is the first two frames of the frame, that is, the nth-2 frame and the nth-1 frame.
Based on any of the above embodiments, step 122 specifically includes: and the current characteristics and the frame front characteristics are fused and then input into the expression prediction layer, so that a virtual expression image of the frame output by the expression prediction layer is obtained.
In the embodiment of the invention, the current characteristic and the frame characteristic of any frame are used for expression prediction, so that the synthesized avatar expression can not only be naturally matched with the voice data corresponding to the frame, but also realize the natural transition of the avatar expression of the frame and the avatar expressions of the previous frames, and further improve the reality and naturalness of the avatar video.
Based on any of the above embodiments, the expression prediction layer includes a candidate expression prediction layer, an optical flow prediction layer, and a fusion layer; fig. 4 is a flowchart of an expression prediction method according to an embodiment of the present invention, as shown in fig. 4, step 122 specifically includes:
step 1221, inputting the fused current feature and the pre-frame feature into a candidate expression prediction layer of the expression prediction layer to obtain a candidate expression map output by the candidate expression prediction layer.
Here, the candidate expression prediction layer is configured to predict an avatar expression of any frame based on a current feature and a pre-frame feature corresponding to the frame, and output a candidate expression map of the frame. Here, the candidate emoticons of the frame are virtual pictograms configured with emotions corresponding to the voice data of the frame.
Step 1222, merging the current feature and the pre-frame feature, and inputting the merged current feature and the pre-frame feature into an optical flow prediction layer of the expression prediction layer to obtain optical flow information output by the optical flow prediction layer.
Here, the optical flow prediction layer is configured to predict an optical flow between a previous frame and a frame based on a current feature and a pre-frame feature corresponding to any frame, and output optical flow information of the frame. Here, the optical flow information of the frame may include the predicted optical flow of the previous frame and the frame, and may further include weights for weighting the optical flow to the candidate emoticons.
Step 1223, inputting the candidate expression image and the optical flow information to a fusion layer in the expression prediction layer, so as to obtain a virtual expression image of the frame output by the fusion layer.
Here, the fusion layer is used for fusing the candidate expression image and the optical flow information of any frame, so as to obtain the virtual expression image of the frame. For example, the fusion layer may directly superimpose the candidate expression map with the virtual expression map of the previous frame after the deformation based on the predicted optical flow, or may superimpose the candidate expression map with the virtual expression map of the previous frame after the deformation based on the predicted optical flow based on the weight obtained by prediction, so as to obtain the virtual expression map.
According to the method provided by the embodiment of the invention, the current characteristics and the pre-frame characteristics are used for carrying out optical flow prediction, and the optical flow information is applied to the generation of the virtual expression graph, so that the muscle movement of each area of the virtual image executing expression in the virtual image video is more natural.
Based on any of the above embodiments, fig. 5 is a schematic structural diagram of an expression synthesis model provided in an embodiment of the present invention, and in fig. 5, the expression synthesis model includes a current feature extraction layer, a pre-frame feature extraction layer, a candidate expression prediction layer, an optical flow prediction layer, and a fusion layer.
The current feature extraction layer is used for obtaining the current features of any frame based on the image data and the related features respectively corresponding to the frame.
Assuming that the relevant feature of the voice data is M, sending M into the long and short-term memory network LSTM may obtain the hidden layer feature HT of the relevant feature, and the hidden layer feature corresponding to each frame may be labeled HT (0), HT (1), …, HT (t), …, and HT (N-1). Wherein HT (t) represents hidden layer characteristics of relevant characteristics corresponding to a t frame, and N is the total frame number of the image data. T frame pairThe corresponding image data includes I (0) and I m (t). Wherein I (O) represents a texture map, I m And (t) represents an expression mask corresponding to the t frame.
In FIG. 5, in the current feature extraction layer, I (O) and I m (t) feeding the first layer convolution (kenerl=3, stride=2, channel_out=64), feeding the obtained feature map into the second layer convolution (kenerl=3, stride=2, channel_out=128), feeding the obtained feature map into the third layer convolution (kenerl=3, stride=2, channel_out=256), feeding the obtained feature map into the fourth layer convolution (kenerl=3, stride=2, channel_out=512), obtaining a 512-dimensional feature map, and then passing through 5 layers of the feature map (kenerl=3, stride=1, channel_out=512) obtaining a 512-dimensional feature map. In the process, hidden layer features HT (t) of the related features are expanded and then embedded into second, third and fourth layers of convolutions and added with convolution results, so that fusion of the related features and image data is realized, and the current feature CFT (t) of a t-th frame is obtained.
In the current feature extraction layer, HT (t) is combined with I (0) and I m When the convolution results FT (t) of (t) are added, only HT (t) is superimposed on the mask region of FT (t), where mask regions, i.e., regions in the avatar where expressions are performed, are superimposed, and non-mask regions of FT (t) are not superimposed. Thus, the expression-related features are superimposed only in the region where the expression is required to be performed, and the original avatar is maintained in the region where the expression is not required to be performed, specifically expressed as the following formula:
in the formula, θ is a relevant parameter of the current feature extraction layer.
The frame front feature extraction layer is used for obtaining frame front features of any frame based on a virtual expression image of a preset frame before the frame.
The pre-set frame preceding the frame is assumed to be the first two frames, namely the t-1 st frame and the t-2 nd frame. The virtual emoticons of the pre-set frame before the frame are Fake (t-1) and Fake (t-2). In the pre-frame feature extraction layer, fake (t-1) and Fake (t-2) are sent to a 4-layer convolutional network (kenerl=3, stride=2, channel_out= 64,128,256,512), and then a 512-dimensional feature map, namely a pre-frame feature PFT (t), is obtained through a 5-layer ResBlock (kenerl=3, stride=1, channel_out=512).
The frame characteristics of the t-th frame are thus obtained, denoted CFT (t) +pft (t).
The candidate expression prediction layer is used for determining a corresponding candidate expression graph according to the input frame characteristics. In the candidate expression prediction layers, the frame characteristics CFT (t) +pft (t) can obtain a candidate expression map of the t frame through 4 layers of ResBlock (kenerl=3, stride=1, channel_out=512) and 4 layers of upsampling layers (kenerl=3, stride=2, channel_out= 256,128,64,1), and the candidate expression map is denoted as S (t). The formula is as follows:
Wherein,is a parameter of the candidate expression prediction layer.
The optical flow prediction layer is used for predicting the optical flow between the previous frame and the frame according to the input frame characteristics and outputting the optical flow information of the frame. In the optical flow prediction layer, the frame characteristics CFT (t) +pft (t) pass through 4 layers of ResBlock (kenerl=3, stride=1, channel_out=512) and 4 layers of upsampling layers (kenerl=3, stride=2, channel_out= 256,128,64,3) to obtain an optical flow F (t-1) and a weighting matrix W (t) between the virtual expression images Fake (t-1) of the previous frame and the virtual expression image Fake (t) of the frame.
The fusion layer is used for fusing the candidate expression image S (t) of any frame with the optical flow information F (t-1) and the optical flow information W (t) to obtain a virtual expression image Fake (t) of the frame. Specifically, the candidate expression map S (t) and the virtual expression map F (t-1) of the previous frame which is deformed by the optical flow F (t-1) can be weighted and summed by the weighting matrix W (t), so that the fusion of the candidate expression map S (t) and the virtual expression map F (t-1) is realized, and the specific formula is as follows:
Fake(t)＝S(t)*W(t)+(1-W(t))*F(t-1)⊙Fake(t-1)
wherein, as follows, the image is deformed by using the optical flow, W (t) is the corresponding weight of the candidate expression map, and 1-W (t) is the corresponding weight of the virtual expression map of the previous frame after the optical flow deformation.
The expression synthesis model provided by the embodiment of the invention can be used for describing the synthesis details of different people under different emotions more vividly through the application of the related characteristics and the integral modeling of the expression, and simultaneously avoids the problem of incompatibility caused by independent synthesis. In addition, the inter-frame continuity of the composite avatar is optimized by the optical flow information.
Based on any embodiment, in the method, the expression synthesis model is obtained based on the sample speaker video, the relevant characteristics of the sample voice data corresponding to the sample speaker video and the sample image data, and the discriminant training, and the expression synthesis model and the discriminant form a generated type countermeasure network.
Specifically, the generated countermeasure network (GAN, generative Adversarial Networks) is a deep learning model, and is one of the most promising methods for unsupervised learning on complex distribution. The generation type antagonism network passes through two modules in the framework: the mutual game learning of the Generative Model and the discriminant Model Discriminative Model produces a fairly good output. In the embodiment of the invention, the expression synthesis model is a generated model, and the discriminant is a discriminant model.
The expression synthesis model is used for synthesizing continuous virtual image videos, and the discriminator is used for discriminating whether the input video is the virtual image video synthesized by the expression synthesis model or the truly recorded video. The role of the discriminator is to judge whether the virtual image video synthesized by the expression synthesis model is true and realistic.
According to the method provided by the embodiment of the invention, through the mutual game learning training of the expression synthesis model and the discriminator, the training effect of the expression synthesis model can be obviously improved, so that the fidelity and naturalness of the virtual image video output by the expression synthesis model can be effectively improved.
Based on any of the above embodiments, the arbiter comprises an image arbiter and/or a video arbiter; the image discriminator is used for judging the synthesis authenticity of any frame of virtual expression graph in the virtual image video, and the video discriminator is used for judging the synthesis authenticity of the virtual image video.
Specifically, the generated countermeasure network may include only an image discriminator or a video discriminator, or may include both the image discriminator and the video discriminator.
The image discriminator is used for judging the authenticity from the image level, namely judging whether the synthesis of the expression, such as the synthesis of the facial and neck muscles, is realistic. The image discriminator can obtain a virtual expression image Fake (t) of the current frame synthesized by the expression synthesis model, send the virtual expression image Fake (t) into a 4-layer convolution network (kenerl=3, stride=1 and channel_out= 64,128,256,1), and calculate an L2 norm between a feature image obtained by convolution and a full 0 matrix with the same size. Similarly, the image discriminator can send any image frame Real (t) in the Real recorded video into the 4-layer convolution network, and calculate the L2 norm between the feature map obtained by convolution and the full 1 matrix with the same size. Here, the all 0 matrix corresponds to a synthesized image, the all 1 matrix corresponds to a true image, and the L2 norm is a loss value of the image discriminator. In order to make the quality of the synthesized virtual expression image higher in each resolution, the virtual expression image output by the expression synthesis model can be respectively sampled by 2 times and 4 times for discrimination.
The video discriminator is used for judging the authenticity at the video level, namely judging whether the synthesis of the video, such as the linkage of the facial and neck muscle movements, is authentic. Multiple continuous virtual expression images and corresponding optical flows synthesized by the expression synthesis model, such as Fake (t-2), fake (t-1), fake (t) and F (t-2) and F (t-1), can be obtained, and can be sent to a video discriminator formed by a 4-layer convolution network (kenerl=3, stride=1 and channel_out= 64,128,256,1), so as to calculate discrimination loss. Similarly, the video discriminator also needs to calculate discrimination loss of the true recorded video. In order to make the quality of the synthesized virtual image video higher in each resolution, the virtual expression images output by the expression synthesis model can be respectively sampled by 2 times and 4 times for discrimination.
In the training process of the expression synthesis model, the opposite loss function of the discriminator can be added into the loss function of the expression synthesis model, so that the expression synthesis model and the discriminator are combined to form countermeasures.
Based on any of the above embodiments, the method wherein the relevant features include language-related features, as well as emotional features and/or speaker identity features.
Wherein the language-dependent features correspond to different pronunciations that require the speaker to mobilize facial muscles to form different mouth shapes, and the facial muscles corresponding to the different mouth shapes are different from the movements of the neck muscles. The emotional features are used to characterize the emotion of the speaker, who speaks the same content under different emotions, and the movements of the facial muscles, including mouth shape, and neck muscles are also different. The identity feature of the speaker is used for representing the identity of the speaker, and specifically may be an identifier corresponding to the speaker, or an identifier corresponding to the occupation of the speaker, or an identifier corresponding to the personality feature, the language style feature, and the like of the speaker.
Based on any of the above embodiments, in the method, the avatar data is determined based on a speaker identity feature.
Specifically, among the mass avatar data stored in advance, different avatar data correspond to different avatars having different identity characteristics. After the speaker identity characteristics in the related characteristics of the voice data are known, the image data matched with the speaker identity characteristics can be selected from the massive image data and applied to the synthesis of the virtual image video.
For example, image data of four A, B, C, D persons is stored in advance. When the speaker identity characteristics of the voice data are known to point to B, the avatar data of B may be correspondingly determined for the synthesis of the avatar video.
Based on any of the above embodiments, step 110 specifically includes: determining acoustic features of the speech data; relevant features are determined based on the acoustic features.
Specifically, the acoustic features herein may be spectrogram and fbank features. For example, the speech data may be denoised using an adaptive filter and the audio sample rate and channel unified, here set to 16K, mono, from which the spectrogram and fbank features (frame shift 10ms, window length 1 s) are then extracted.
Thereafter, the BN feature sequences representing the language content can be extracted as language-dependent features using the bottleneck network, respectively, and here set to obtain a 256-dimensional BN feature at intervals of 40ms, denoted as L (0), L (1), …, L (N-1), N being the number of frames of 25fps video. Compared with the method based on the phoneme characteristic in the prior art, the BN characteristic is irrelevant to languages, even if the expression synthesis model is trained only in Chinese, the correct mouth shape can be synthesized by using other languages when the expression synthesis model is trained. In addition, in the embodiment of the invention, the high-dimensional feature sequence for expressing emotion is extracted from the convolutionally long-short-time memory network ConvLSTM which is fully trained on 8 basic expression (lively, happy, afraid, depressed, excited, surprise, sad and neutral) recognition tasks and is used as emotion features. Here, a 128-dimensional emotion vector is set for every 40ms, denoted E (0), E (1), …, E (N-1), N being the number of frames of 25fps video. Similarly, in order to achieve personalized customization, in the embodiment of the invention, a speaker identity recognition network based on a deep neural network DNN and an i-vector is used for extracting a speaker identity feature sequence, wherein the speaker identity feature sequence is set to obtain a 128-dimensional identity feature vector at intervals of 40ms, the 128-dimensional identity feature vector is recorded as P (0), P (1), … and P (N-1), and N is the number of frames of 25fps video. Finally, the three feature sequences are spliced according to the corresponding frames, and 512-dimensional fusion related features are obtained for each frame and are marked as M (0), M (1), … and M (N-1), wherein N is the number of frames of the 25fps video.
Based on any of the above embodiments, in the method, the expression corresponding to the voice data of the avatar configuration in the avatar video includes a facial expression and a neck expression.
Correspondingly, the expression mask map in the image data covers the part comprising an execution area of the facial expression and an execution area of the neck expression. Here, the facial expression execution area may include facial muscle areas such as frontal muscle, orbicularis oculi muscle, frowning muscle, orbicularis stomatitis muscle, and the like, and does not include eyeball areas and nose bridge areas, because the movement of eyeballs is not controlled by facial muscles, and the nose bridge has bones, approximately rigid body, and is little affected by the movement of muscles in other areas of the face.
In the embodiment of the invention, the facial expression and the neck expression are combined to be used as the expression whole, and compared with a mode of singly modeling each region of the expression in the virtual image, the method can effectively solve the problem of the linkage of the muscles of each region by aiming at the expression whole modeling, so that the muscle linkage of each region is more natural and lifelike.
Based on any of the above embodiments, fig. 6 is a schematic flow chart of an avatar composition method according to another embodiment of the present invention, as shown in fig. 6, the method includes:
Step 610, determining voice data:
extracting voice data from the collected video and audio data, denoising the voice data by using an adaptive filter, unifying an audio sampling rate and a sound channel, and then extracting a spectrogram and fbank characteristics from the voice data to be identified. In order to fully ensure the time sequence of the voice data, the embodiment of the invention does not need to split the input voice data.
Step 620, acquiring relevant features of the voice data:
and (3) respectively obtaining the language-related features, the emotion features and the speaker identity features corresponding to the voice data to be recognized of each frame through a neural network for extracting the language-related features, the emotion features and the speaker identity features of the voice data obtained in the previous step, and splicing the three features according to the corresponding frames to obtain the corresponding related features of each frame.
Step 630, determining video data, detecting a face area, and cutting a head area:
extracting video data from the collected video and audio data, detecting a face area of each frame of image, taking the obtained face frame as a reference, expanding the face area outwards by 1.5 times to obtain an area containing the whole head and neck, cutting the area, storing the area as an image sequence, and recording as I (0), I (1), …, I (N-1) and N as the frame number of 25fps video.
Step 640, generating image data:
facial muscle areas and neck muscle areas such as frontal muscle, orbicularis oculi, frowning muscle, orbicularis stomatalis muscle and the like of each frame of cut image I (t) are segmented according to skin colors and physiological structural characteristics of a human face or by using a neural network, and eyeball areas and nose bridge areas are not included, so that the movement of eyeballs is not controlled by facial muscles, and the nose bridge has bones, is approximately rigid, and is little influenced by the movement of muscles of other areas of the face. The pixel values of the facial muscle region and the neck muscle region are set to zero to obtain an expression mask image sequence, which is marked as Im (0), im (1), …, im (N-1), and N is the number of frames of the video of 25 fps.
The image data thus obtained contains a texture map I (0) and an expression mask map corresponding to each frame.
Step 650, inputting an expression synthesis model to obtain an avatar head video:
in the expression synthesis model, a feature map is obtained from a texture map and an expression mask map in image data through a plurality of layers of convolution networks, the feature map is fused with spliced related features, then face and neck regions are synthesized through a plurality of layers of convolution networks, and finally optical flow information is added into a video, so that the synthesized mouth shape, expression, throat movement and the like are more natural.
For example, the texture map entered is expressionless and the speech data is excited to say "abstract-! For irrelevant areas, such as the hair, nose and other areas of the texture map, the relevant areas, such as the mouth shape, cheek, eyebrow and other areas, the original areas are deformed into new textures according to relevant characteristics and texture images, and the final synthesized virtual expression map is obtained through fusion.
Step 660, merging the avatar header video and the body part of the video data:
if the head area of the synthesized virtual head image is spliced into the video according to the original coordinates, tiny joints appear at the boundary, and preferably, a poisson fusion algorithm can be used for fusing the joint areas, so that the boundary transition is smoother.
Compared with the traditional virtual image synthesis technology based on voice driving and the face synthesis technology based on expression migration, the method provided by the embodiment of the invention can more realistically synthesize the facial and neck muscle movements of different people under different emotions, and can realize full-automatic offline synthesis. Saving a large amount of labor cost and improving the production efficiency.
Based on any one of the above embodiments, fig. 7 is a schematic structural view of an avatar composition device according to an embodiment of the present invention, and as shown in fig. 7, the device includes a relevant feature determining unit 710 and an expression composition unit 720;
wherein the relevant feature determining unit 710 is configured to determine relevant features of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data;
the expression synthesis unit 720 is configured to input the avatar data and the related features into an expression synthesis model, so as to obtain an avatar video output by the expression synthesis model, where an avatar in the avatar video is configured with an expression corresponding to the voice data;
the expression synthesis model is obtained by training relevant features of sample voice data and sample image data corresponding to a sample speaker video.
The device provided by the embodiment of the invention synthesizes the expression of the virtual image by applying the related characteristics containing rich expression related information, so that the expression of the virtual image can be better attached to the voice data, and the device is more natural and real. In addition, in the virtual image video generated by the expression synthesis model, the expression of the virtual image exists in an integral form, compared with a mode of singly modeling each region of the expression in the virtual image, the method can effectively solve the problem of the linkage of the muscles of each region by aiming at the integral modeling of the expression, so that the muscle linkage of each region is more natural and lifelike.
Based on any of the above embodiments, the expression synthesis unit 720 includes:
the feature extraction unit is used for inputting the image data and the related features corresponding to any frame to a feature extraction layer of the expression synthesis model to obtain frame features output by the feature extraction layer;
and the expression prediction unit is used for inputting the frame characteristics to an expression prediction layer of the expression synthesis model to obtain a virtual expression image of any frame output by the expression prediction layer.
Based on any of the above embodiments, the feature extraction unit includes:
a current feature extraction subunit, configured to input image data and related features corresponding to any frame respectively to a current feature extraction layer of the feature extraction layer, so as to obtain a current feature output by the current feature extraction layer;
and the pre-frame feature extraction subunit is used for inputting the virtual expression image of any pre-frame preset frame to the pre-frame feature extraction layer of the feature extraction layer to obtain the pre-frame features output by the pre-frame feature extraction layer.
Based on any of the above embodiments, the expression prediction unit is specifically configured to:
and the current features and the pre-frame features are fused and then input into the expression prediction layer, so that a virtual expression image of any frame output by the expression prediction layer is obtained.
Based on any of the above embodiments, the expression prediction unit includes:
the candidate expression prediction subunit is used for inputting the current characteristics and the frame front characteristics into a candidate expression prediction layer of the expression prediction layer after fusing, so as to obtain a candidate expression image output by the candidate expression prediction layer;
the optical flow prediction subunit is used for inputting the current feature and the frame front feature into the optical flow prediction layer of the expression prediction layer after fusing, so as to obtain optical flow information output by the optical flow prediction layer;
and the fusion subunit is used for inputting the candidate expression images and the optical flow information into a fusion layer in the expression prediction layer to obtain the virtual expression image of any frame output by the fusion layer.
Based on any one of the above embodiments, the expression synthesis model is obtained by training based on a sample speaker video, relevant features of sample voice data corresponding to the sample speaker video, sample image data, and a discriminator, where the expression synthesis model and the discriminator form a generated type countermeasure network.
Based on any of the above embodiments, the arbiter comprises an image arbiter and/or a video arbiter;
the image discriminator is used for judging the synthesis authenticity of any frame of virtual expression graph in the virtual image video, and the video discriminator is used for judging the synthesis authenticity of the virtual image video.
Based on any of the above embodiments, the relevant features include language-related features, as well as emotional features and/or speaker identity features.
Based on any of the above embodiments, the persona data is determined based on the speaker identity characteristics.
Based on any of the above embodiments, the expression of the avatar configuration in the avatar video corresponding to the voice data includes a facial expression and a neck expression.
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 8, the electronic device may include: processor 810, communication interface (Communications Interface) 820, memory 830, and communication bus 840, wherein processor 810, communication interface 820, memory 830 accomplish communication with each other through communication bus 840. The processor 810 may call logic instructions in the memory 830 to perform the following method: determining relevant characteristics of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data; inputting the image data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with expressions corresponding to the voice data; the expression synthesis model is obtained by training relevant features of sample voice data and sample image data corresponding to a sample speaker video.
Further, the logic instructions in the memory 830 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Embodiments of the present invention also provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the methods provided by the above embodiments, for example, comprising: determining relevant characteristics of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data; inputting the image data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with expressions corresponding to the voice data; the expression synthesis model is obtained by training relevant features of sample voice data and sample image data corresponding to a sample speaker video.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A avatar composition method, comprising:
determining relevant characteristics of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data;
inputting the image data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with expressions corresponding to the voice data;
the expression synthesis model is obtained by training based on a sample speaker video, relevant characteristics of sample voice data corresponding to the sample speaker video and sample image data;
Inputting the image data and the related features into an expression synthesis model to obtain an virtual image video output by the expression synthesis model, wherein the virtual image video comprises the following specific steps:
inputting image data and related features corresponding to any frame in the voice data to a feature extraction layer of the expression synthesis model respectively to obtain frame features output by the feature extraction layer, wherein the frame features comprise current features and pre-frame features;
inputting the frame characteristics to an expression prediction layer of the expression synthesis model to obtain a virtual expression image of any frame output by the expression prediction layer, wherein the virtual expression image is a frame of image containing an avatar, and each frame of virtual expression image forms an avatar video;
the step of inputting the frame characteristics to an expression prediction layer of the expression synthesis model to obtain a virtual expression map of any frame output by the expression prediction layer, comprises the following steps:
the current characteristics and the frame front characteristics are fused and then input into a candidate expression prediction layer of the expression prediction layer, so that a candidate expression image output by the candidate expression prediction layer is obtained;
the current features and the frame front features are fused and then input into an optical flow prediction layer of the expression prediction layer, so that optical flow information output by the optical flow prediction layer is obtained;
And inputting the candidate expression images and the optical flow information into a fusion layer in the expression prediction layer to obtain a virtual expression image of any frame output by the fusion layer.
2. The avatar composition method as claimed in claim 1, wherein the inputting of the avatar data and related features corresponding to any frame of the voice data to the feature extraction layer of the expression composition model, respectively, obtains frame features outputted from the feature extraction layer, comprises:
inputting the image data and related features corresponding to any frame to a current feature extraction layer of the feature extraction layer to obtain current features output by the current feature extraction layer;
and inputting the virtual expression image of any frame preset before to a frame front feature extraction layer of the feature extraction layer to obtain the frame front features output by the frame front feature extraction layer.
3. The avatar composition method of claim 1, wherein the expression composition model is based on a sample speaker video, correlation features of sample voice data corresponding to the sample speaker video and sample avatar data, and a discriminant training, the expression composition model and the discriminant constituting a generative countermeasure network.
4. A method of avatar composition according to claim 3, wherein the discriminator comprises an image discriminator and/or a video discriminator;
the image discriminator is used for judging the synthesis authenticity of any frame of virtual expression graph in the virtual image video, and the video discriminator is used for judging the synthesis authenticity of the virtual image video.
5. The avatar composition method of any one of claims 1 to 4, wherein the related features include language related features, and emotional and/or speaker identity features.
6. The avatar composition method of claim 5, wherein the avatar data is determined based on the speaker identity characteristics.
7. The avatar composition method of any one of claims 1 to 4, wherein the expression corresponding to the voice data of the avatar configuration in the avatar video includes a facial expression and a neck expression.
8. An avatar composition device, comprising:
a relevant feature determining unit for determining relevant features of the voice data; the relevant features are used for representing the features related to the expression of the speaker contained in the voice data;
The expression synthesis unit is used for inputting the image data and the related features into an expression synthesis model to obtain an avatar video output by the expression synthesis model, wherein the avatar in the avatar video is configured with an expression corresponding to the voice data;
the expression synthesis model is obtained by training relevant features of sample voice data and sample image data corresponding to a sample speaker video;
the expression synthesis unit is specifically configured to:
inputting image data and related features corresponding to any frame in the voice data to a feature extraction layer of the expression synthesis model respectively to obtain frame features output by the feature extraction layer, wherein the frame features comprise current features and pre-frame features;
inputting the frame characteristics to an expression prediction layer of the expression synthesis model to obtain a virtual expression image of any frame output by the expression prediction layer, wherein the virtual expression image is a frame of image containing an avatar, and each frame of virtual expression image forms an avatar video;
the current characteristics and the pre-frame characteristics are fused and then input into a candidate expression prediction layer of the expression prediction layer, so that a candidate expression image output by the candidate expression prediction layer is obtained;
The current features and the frame front features are fused and then input into an optical flow prediction layer of the expression prediction layer, so that optical flow information output by the optical flow prediction layer is obtained;
and inputting the candidate expression images and the optical flow information into a fusion layer in the expression prediction layer to obtain a virtual expression image of any frame output by the fusion layer.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the avatar composition method as claimed in any one of claims 1 to 7 when the program is executed.
10. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of the avatar composition method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911274701.3A CN111145282B (en) | 2019-12-12 | 2019-12-12 | Avatar composition method, apparatus, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911274701.3A CN111145282B (en) | 2019-12-12 | 2019-12-12 | Avatar composition method, apparatus, electronic device, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111145282A CN111145282A (en) | 2020-05-12 |
| CN111145282B true CN111145282B (en) | 2023-12-05 |
Family
ID=70518080
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911274701.3A Active CN111145282B (en) | 2019-12-12 | 2019-12-12 | Avatar composition method, apparatus, electronic device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111145282B (en) |
Families Citing this family (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111915479B (en) * | 2020-07-15 | 2024-04-26 | 抖音视界有限公司 | Image processing method and device, electronic equipment and computer readable storage medium |
| CN112132915B (en) * | 2020-08-10 | 2022-04-26 | 浙江大学 | Diversified dynamic time-delay video generation method based on generation countermeasure mechanism |
| CN112215927B (en) * | 2020-09-18 | 2023-06-23 | 腾讯科技（深圳）有限公司 | Face video synthesis method, device, equipment and medium |
| CN112182173B (en) * | 2020-09-23 | 2024-08-06 | 支付宝(杭州)信息技术有限公司 | Man-machine interaction method and device based on virtual life and electronic equipment |
| CN112465935A (en) * | 2020-11-19 | 2021-03-09 | 科大讯飞股份有限公司 | Virtual image synthesis method and device, electronic equipment and storage medium |
| CN112492383A (en) * | 2020-12-03 | 2021-03-12 | 珠海格力电器股份有限公司 | Video frame generation method and device, storage medium and electronic equipment |
| CN112650399B (en) * | 2020-12-22 | 2023-12-01 | 科大讯飞股份有限公司 | Expression recommendation method and device |
| CN114793286A (en) * | 2021-01-25 | 2022-07-26 | 上海哔哩哔哩科技有限公司 | Video editing method and system based on virtual image |
| CN114793300A (en) * | 2021-01-25 | 2022-07-26 | 天津大学 | Virtual video customer service robot synthesis method and system based on generation countermeasure network |
| CN112785669B (en) * | 2021-02-01 | 2024-04-23 | 北京字节跳动网络技术有限公司 | Virtual image synthesis method, device, equipment and storage medium |
| CN113096242A (en) * | 2021-04-29 | 2021-07-09 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, electronic equipment and storage medium |
| WO2022255980A1 (en) * | 2021-06-02 | 2022-12-08 | Bahcesehir Universitesi | Virtual agent synthesis method with audio to video conversion |
| CN114466179B (en) * | 2021-09-09 | 2024-09-06 | 马上消费金融股份有限公司 | Method and device for measuring synchronism of voice and image |
| CN114911381B (en) * | 2022-04-15 | 2023-06-16 | 青岛海尔科技有限公司 | Interactive feedback method and device, storage medium and electronic device |
| CN114937104B (en) * | 2022-06-24 | 2024-08-13 | 北京有竹居网络技术有限公司 | Virtual object face information generation method and device and electronic equipment |
| CN115375809B (en) * | 2022-10-25 | 2023-03-14 | 科大讯飞股份有限公司 | Method, device and equipment for generating virtual image and storage medium |
| CN116665695B (en) * | 2023-07-28 | 2023-10-20 | 腾讯科技（深圳）有限公司 | Virtual object mouth shape driving method, related device and medium |
| CN117221465B (en) * | 2023-09-20 | 2024-04-16 | 北京约来健康科技有限公司 | Digital video content synthesis method and system |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106919251A (en) * | 2017-01-09 | 2017-07-04 | 重庆邮电大学 | A kind of collaborative virtual learning environment natural interactive method based on multi-modal emotion recognition |
| CN107705808A (en) * | 2017-11-20 | 2018-02-16 | 合光正锦（盘锦）机器人技术有限公司 | A kind of Emotion identification method based on facial characteristics and phonetic feature |
| WO2018113650A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳市掌网科技股份有限公司 | Virtual reality language interaction system and method |
| CN108989705A (en) * | 2018-08-31 | 2018-12-11 | 百度在线网络技术（北京）有限公司 | A kind of video creating method of virtual image, device and terminal |
| CN109118562A (en) * | 2018-08-31 | 2019-01-01 | 百度在线网络技术（北京）有限公司 | Explanation video creating method, device and the terminal of virtual image |
| CN109145837A (en) * | 2018-08-28 | 2019-01-04 | 厦门理工学院 | Face emotion identification method, device, terminal device and storage medium |
| CN109410297A (en) * | 2018-09-14 | 2019-03-01 | 重庆爱奇艺智能科技有限公司 | It is a kind of for generating the method and apparatus of avatar image |
| CN110009716A (en) * | 2019-03-28 | 2019-07-12 | 网易（杭州）网络有限公司 | Generation method, device, electronic equipment and the storage medium of facial expression |
| CN110414323A (en) * | 2019-06-14 | 2019-11-05 | 平安科技（深圳）有限公司 | Mood detection method, device, electronic equipment and storage medium |
| CN110488975A (en) * | 2019-08-19 | 2019-11-22 | 深圳市仝智科技有限公司 | A kind of data processing method and relevant apparatus based on artificial intelligence |
| CN110503942A (en) * | 2019-08-29 | 2019-11-26 | 腾讯科技（深圳）有限公司 | A kind of voice driven animation method and device based on artificial intelligence |
-
2019
- 2019-12-12 CN CN201911274701.3A patent/CN111145282B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018113650A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳市掌网科技股份有限公司 | Virtual reality language interaction system and method |
| CN106919251A (en) * | 2017-01-09 | 2017-07-04 | 重庆邮电大学 | A kind of collaborative virtual learning environment natural interactive method based on multi-modal emotion recognition |
| CN107705808A (en) * | 2017-11-20 | 2018-02-16 | 合光正锦（盘锦）机器人技术有限公司 | A kind of Emotion identification method based on facial characteristics and phonetic feature |
| CN109145837A (en) * | 2018-08-28 | 2019-01-04 | 厦门理工学院 | Face emotion identification method, device, terminal device and storage medium |
| CN108989705A (en) * | 2018-08-31 | 2018-12-11 | 百度在线网络技术（北京）有限公司 | A kind of video creating method of virtual image, device and terminal |
| CN109118562A (en) * | 2018-08-31 | 2019-01-01 | 百度在线网络技术（北京）有限公司 | Explanation video creating method, device and the terminal of virtual image |
| CN109410297A (en) * | 2018-09-14 | 2019-03-01 | 重庆爱奇艺智能科技有限公司 | It is a kind of for generating the method and apparatus of avatar image |
| CN110009716A (en) * | 2019-03-28 | 2019-07-12 | 网易（杭州）网络有限公司 | Generation method, device, electronic equipment and the storage medium of facial expression |
| CN110414323A (en) * | 2019-06-14 | 2019-11-05 | 平安科技（深圳）有限公司 | Mood detection method, device, electronic equipment and storage medium |
| CN110488975A (en) * | 2019-08-19 | 2019-11-22 | 深圳市仝智科技有限公司 | A kind of data processing method and relevant apparatus based on artificial intelligence |
| CN110503942A (en) * | 2019-08-29 | 2019-11-26 | 腾讯科技（深圳）有限公司 | A kind of voice driven animation method and device based on artificial intelligence |
Non-Patent Citations (3)
| Title |
|---|
| Speech and Auditory Interfaces for Ubiquitous, Immersive and Personalized Applications;Lei Xie等;《 IEEE Xplore》;全文 * |
| 基于特征流的面部表情运动分析及应用;金辉等;《软件学报》(第12期);全文 * |
| 语音驱动的人脸动画研究现状综述;李欣怡等;《计算机工程与应用》(第22期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111145282A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111145282B (en) | Avatar composition method, apparatus, electronic device, and storage medium | |
| CN111145322B (en) | Method, apparatus, and computer-readable storage medium for driving avatar | |
| Busso et al. | Rigid head motion in expressive speech animation: Analysis and synthesis | |
| US8224652B2 (en) | Speech and text driven HMM-based body animation synthesis | |
| CN112465935A (en) | Virtual image synthesis method and device, electronic equipment and storage medium | |
| CN113781610B (en) | Virtual face generation method | |
| US7136818B1 (en) | System and method of providing conversational visual prosody for talking heads | |
| CN113454708A (en) | Linguistic style matching agent | |
| CN110610534B (en) | Automatic mouth shape animation generation method based on Actor-Critic algorithm | |
| US20120130717A1 (en) | Real-time Animation for an Expressive Avatar | |
| KR102509666B1 (en) | Real-time face replay based on text and audio | |
| Mariooryad et al. | Generating human-like behaviors using joint, speech-driven models for conversational agents | |
| US20110131041A1 (en) | Systems And Methods For Synthesis Of Motion For Animation Of Virtual Heads/Characters Via Voice Processing In Portable Devices | |
| EP3915108B1 (en) | Real-time generation of speech animation | |
| US11989976B2 (en) | Nonverbal information generation apparatus, nonverbal information generation model learning apparatus, methods, and programs | |
| WO2023284435A1 (en) | Method and apparatus for generating animation | |
| KR102373608B1 (en) | Electronic apparatus and method for digital human image formation, and program stored in computer readable medium performing the same | |
| US20210005218A1 (en) | Nonverbal information generation apparatus, method, and program | |
| JPWO2019160105A1 (en) | Non-verbal information generator, non-verbal information generation model learning device, method, and program | |
| Filntisis et al. | Video-realistic expressive audio-visual speech synthesis for the Greek language | |
| JP2015038725A (en) | Utterance animation generation device, method, and program | |
| US20210370519A1 (en) | Nonverbal information generation apparatus, nonverbal information generation model learning apparatus, methods, and programs | |
| Verma et al. | Animating expressive faces across languages | |
| Filntisis et al. | Photorealistic adaptation and interpolation of facial expressions using HMMS and AAMS for audio-visual speech synthesis | |
| Chandrasiri et al. | Internet communication using real-time facial expression analysis and synthesis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |