CN111914106B - Texture and normal library construction method, texture and normal map generation method and device - Google Patents
Texture and normal library construction method, texture and normal map generation method and device Download PDFInfo
- Publication number
- CN111914106B CN111914106B CN202010837186.1A CN202010837186A CN111914106B CN 111914106 B CN111914106 B CN 111914106B CN 202010837186 A CN202010837186 A CN 202010837186A CN 111914106 B CN111914106 B CN 111914106B
- Authority
- CN
- China
- Prior art keywords
- face
- texture
- principal component
- face texture
- texture map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000010276 construction Methods 0.000 title claims abstract description 88
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000005070 sampling Methods 0.000 claims abstract description 82
- 238000000513 principal component analysis Methods 0.000 claims abstract description 71
- 239000013598 vector Substances 0.000 claims description 108
- 238000012545 processing Methods 0.000 claims description 50
- 238000006243 chemical reaction Methods 0.000 claims description 15
- 238000012163 sequencing technique Methods 0.000 claims description 13
- 238000013519 translation Methods 0.000 claims description 11
- 238000005516 engineering process Methods 0.000 abstract description 20
- 238000013473 artificial intelligence Methods 0.000 abstract description 16
- 230000002708 enhancing effect Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 28
- 239000011159 matrix material Substances 0.000 description 24
- 230000008569 process Effects 0.000 description 17
- 238000000354 decomposition reaction Methods 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 12
- 238000013461 design Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 8
- 210000001508 eye Anatomy 0.000 description 8
- 238000010801 machine learning Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 210000004709 eyebrow Anatomy 0.000 description 6
- 210000001331 nose Anatomy 0.000 description 5
- 230000015572 biosynthetic process Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000003786 synthesis reaction Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 210000000887 face Anatomy 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 208000025174 PANDAS Diseases 0.000 description 2
- 208000021155 Paediatric autoimmune neuropsychiatric disorders associated with streptococcal infection Diseases 0.000 description 2
- 240000004718 Panda Species 0.000 description 2
- 235000016496 Panda oleosa Nutrition 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000008921 facial expression Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 210000001061 forehead Anatomy 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000007493 shaping process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 238000013316 zoning Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5862—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Library & Information Science (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Computer Graphics (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Generation (AREA)
Abstract
The application discloses a texture and normal library construction method based on artificial intelligence technology, which comprises the following steps: acquiring a first original face texture map set; generating a first sampling face texture map set according to the first original face texture map set; determining a first principal component analysis parameter according to a first sampling face texture map set aiming at a target face region; aiming at a target face area, if the construction condition of the face texture and the normal line library is met, the first principal component analysis parameter is used as a texture normal line base corresponding to the target face area in the face texture and the normal line library, and the face texture and the normal line library also comprise a texture normal line base corresponding to the (K-1) face area. The embodiment of the application also provides a method and a device for generating the texture and normal map, which can construct a new face texture and normal library, and can restore the face texture map with lower definition into the ultra-high definition face texture map by using the face texture and normal library, thereby enhancing the expression of the face texture map.
Description
Technical Field
The application relates to the field of computer vision, in particular to a texture and normal library construction method, a texture and normal map generation method and a texture and normal map generation device.
Background
With the development of deep learning, three-dimensional faces play an increasingly important role in many fields. Generating three-dimensional face models is a powerful tool in Computer Vision (CV) that achieves invariance of pose and illumination by modeling three-dimensional face space and imaging processes.
The existing open source face texture library comprises a Dynamic 3D face motion coding system data set (Dynamic 3D Facial Action Coding System Dataset,D3DFACS), and when a D3DFACS is constructed, a series of faces and corresponding texture information thereof are obtained through scanning by high-precision scanning equipment or low-precision scanning equipment.
However, the texture map in the face texture library usually has a smaller size, for example, the size of the face texture map provided in D3DFACS is 1024×1280, and the smaller texture map means that there is a defect in the texture expression, and it is difficult to complete the task of fitting the ultra-high-definition face image.
Disclosure of Invention
The embodiment of the application provides a texture and normal library construction method, a texture and normal map generation method and a texture and normal map generation device, which can construct a new face texture and normal library, and can restore a face texture map with lower definition into an ultra-high definition face texture map by using the face texture and normal library, so that the expression of the face texture map is enhanced, and the task of the ultra-high definition face image can be realized.
In view of this, the present application provides a texture and normal library construction method, which includes:
acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map has K face areas, N is an integer greater than 1, and K is an integer greater than 1;
generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
determining a first principal component analysis parameter according to a first sampling face texture map set aiming at a target face region, wherein the target face region belongs to any face region in the K face regions;
and aiming at the target face region, if the face texture and normal library construction condition is met, taking the first principal component analysis parameter as a texture normal basis corresponding to the target face region in the face texture and normal library, wherein the face texture and normal library also comprises a texture normal basis corresponding to the (K-1) face region.
Another aspect of the present application provides a texture and normal library construction method, including:
acquiring an initial face texture map, wherein the initial face texture map corresponds to a first size;
determining a K face area according to the initial face texture map, wherein K is an integer greater than 1;
for each face region included in the initial face texture map, obtaining target region maps corresponding to the K face regions respectively through a face texture and normal library, wherein the face texture and normal library comprises K groups of texture normal bases, each group of texture normal bases corresponds to one face region, and the face texture and normal library is obtained by adopting the texture and normal library construction method according to any one of claims 1 to 9;
and generating a target face texture map according to the target region maps respectively corresponding to the K face regions, wherein the target face texture map corresponds to a second size which is larger than the first size.
Another aspect of the present application provides a texture and normal library construction apparatus, including:
the acquisition module is used for acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map has K face areas, N is an integer greater than 1, and K is an integer greater than 1;
The generating module is used for generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
the determining module is used for determining a first principal component analysis parameter according to a first sampling face texture map set aiming at a target face area, wherein the target face area belongs to any face area in the K face areas;
the construction module is used for aiming at the target face area, if the construction condition of the face texture and the normal line library is met, the first principal component analysis parameter is used as a texture normal line base corresponding to the target face area in the face texture and the normal line library, wherein the face texture and the normal line library also comprise a texture normal line base corresponding to the (K-1) face area.
In one possible design, in one implementation of another aspect of the embodiments of the present application,
the acquisition module is specifically configured to acquire a second original face texture map set, where the second original face texture map set includes M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
And performing disturbance processing on any one of the second original face texture maps in the second original face texture map set to obtain at least one first original face texture map contained in the first original face texture map set, wherein the disturbance processing comprises at least one of rotation, translation and scaling.
In one possible design, in another implementation of another aspect of the embodiments of the present application,
the acquisition module is specifically used for acquiring M face texture maps, wherein each face texture map comprises data of a red channel, data of a green channel and data of a blue channel;
obtaining M face normal maps, wherein each face normal map comprises data of a red channel, data of a green channel and data of a blue channel, and the face normal maps and the face texture map have a corresponding relation;
and combining each face texture map with the corresponding face normal map to obtain a second original face texture map set, wherein each second original face texture map comprises data of two red channels, data of two green channels and data of two blue channels.
In one possible design, in another implementation of another aspect of the embodiments of the present application,
the acquisition module is further used for acquiring a first high mean value, a first high principal component coefficient and a first high principal component standard deviation according to the first original face texture image set aiming at the target face region after the first original face texture image set is acquired;
the determining module is further used for determining a first high principal component according to the first high principal component coefficient and the first high principal component standard deviation aiming at the target face area;
the determining module is specifically configured to obtain, for a target face area, a first low average value, a first low principal component coefficient, and a first low principal component standard deviation according to a first sampled face texture atlas;
determining a first low principal component according to the first low principal component coefficient and the first low principal component standard deviation for the target face region;
and aiming at the target face area, acquiring a first principal component analysis parameter according to the first high average value, the first low average value, the first high principal component and the first low principal component.
In one possible design, in another implementation of another aspect of the embodiments of the present application,
The determining module is specifically configured to obtain, for a target face area, a first low average value, a first low principal component coefficient, and a first low principal component standard deviation according to a first sampled face texture atlas;
aiming at a target face area, acquiring a first high mean value according to a first original face texture atlas;
acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with a first original face texture map;
for a target face region, determining a first high principal component according to the first low mean value, the first low principal component coefficient, the first low principal component standard deviation, the first high mean value and the first texture fitting coefficient;
and aiming at the target face area, acquiring a first principal component analysis parameter according to the first high average value, the first low average value, the first high principal component and the first low principal component.
In one possible design, in another implementation of another aspect of the embodiments of the present application,
the determining module is specifically configured to perform principal component analysis on the first sampled face texture atlas aiming at the target face area to obtain a first low average value and a low principal component coefficient to be processed, where the low principal component coefficient to be processed includes Q feature vectors, and Q is an integer greater than 1;
Aiming at a target face area, acquiring a low principal component variance to be processed according to the low principal component coefficient to be processed, wherein the low principal component variance to be processed comprises Q characteristic values, and the characteristic values and the characteristic vectors have a corresponding relation;
aiming at a target face area, arranging Q characteristic values included in the variance of the low principal component to be processed according to the sequence from big to small to obtain Q characteristic values after sequencing;
for a target face region, T characteristic values with the characteristic value duty ratio larger than a duty ratio threshold value are obtained from the Q characteristic values after sequencing, wherein T is an integer larger than or equal to 1 and smaller than or equal to Q;
aiming at a target face area, acquiring a first low principal component standard deviation according to T characteristic values;
for a target face region, determining T feature vectors corresponding to the T feature values from Q feature vectors included in the low principal component coefficients to be processed, and acquiring a first low principal component coefficient according to the T feature vectors.
In one possible design, in another implementation of another aspect of the embodiments of the present application,
the acquisition module is further used for acquiring a first texture fitting coefficient after the determination module determines a first principal component analysis parameter according to the first sampling face texture map set aiming at the target face region, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with the first original face texture map;
The determining module is further configured to determine, for the target face area, N first loss values according to the first texture fitting coefficient, the first original face texture map set, and the first high mean value and the first high principal component included in the first principal component analysis parameter, where the first loss values have a correspondence with the first original face texture map;
the determining module is further configured to determine that a face texture and normal library construction condition is met if, for the target face region, a difference value between a maximum loss value of the N first loss values and a maximum loss value of the M second loss values is smaller than or equal to a difference threshold, where the M second loss values are determined according to a second original face texture atlas, and the second original face texture atlas generates a first original face texture atlas after disturbance processing.
In one possible design, in another implementation of another aspect of the embodiments of the present application,
the acquisition module is further used for acquiring a second original face texture map set before acquiring the first original face texture map set, wherein the second original face texture map set comprises M second original face texture maps, each second original face texture map has K face areas, and M is an integer larger than 1;
The generating module is further configured to generate a second sampled face texture map set according to the second original face texture map set, where the second sampled face texture map set includes M second sampled face texture maps, each second sampled face texture map is obtained by downsampling the second original face texture map, and each second sampled face texture map has K face regions;
the determining module is further configured to determine, according to the second sampled face texture map set, a second principal component analysis parameter for the target face region, where the second principal component analysis parameter includes a second high average value, a second low average value, a second high principal component, and a second low principal component;
the acquisition module is further used for acquiring a second texture fitting coefficient, wherein the second texture fitting coefficient comprises M texture vectors, and each texture vector in the M texture vectors has a corresponding relation with a second original face texture map;
the determining module is further configured to determine, for the target face area, M second loss values according to the second texture fitting coefficient, the second original face texture map set, and a second high average value and a second high principal component included in the second principal component analysis parameter, where the second loss values have a correspondence with the second original face texture map.
In one possible design, in another implementation of another aspect of the embodiments of the present application, the texture and normal library construction apparatus further includes a sorting module and a perturbation module;
the sorting module is used for sorting the N first loss values according to the sequence from large to small to obtain sorted N first loss values if the difference value between the maximum loss value in the N first loss values and the maximum loss value in the M second loss values is larger than a difference value threshold value;
the acquisition module is further used for acquiring the first loss values which are arranged in front from the N first loss values which are sequenced according to the target face area, wherein H is an integer which is greater than or equal to 1 and less than or equal to N;
the determining module is further used for determining corresponding H first original face texture graphs according to the H first loss values;
the disturbance module is used for carrying out disturbance processing on the H first original face texture images to obtain at least one third original face texture image, wherein the disturbance processing comprises at least one of rotation, translation and scale scaling.
Another aspect of the present application provides a texture and normal map generating apparatus, including:
The acquisition module is used for acquiring an initial face texture map, wherein the initial face texture map corresponds to a first size;
the determining module is used for determining K face areas according to the initial face texture map, wherein K is an integer greater than 1;
the obtaining module is further configured to obtain, for each face region included in the initial face texture map, a target region map corresponding to each of the K face regions through a face texture and normal library, where the face texture and normal library includes K groups of texture normal bases, each group of texture normal bases corresponds to one face region, and the face texture and normal library is obtained by using the texture and normal library construction method according to any one of claims 1 to 9;
the generating module is used for generating a target face texture map according to the target region maps respectively corresponding to the K face regions, wherein the target face texture map corresponds to a second size, and the second size is larger than the first size.
In one possible design, in one implementation of another aspect of the embodiments of the present application, the texture and normal map generating apparatus further includes a presentation module;
the acquisition module is also used for acquiring an image selection instruction through the terminal equipment, wherein the image selection instruction is used for indicating a target conversion image;
The acquisition module is further used for acquiring a target conversion image from an image set to be converted according to the image selection instruction, wherein the image set to be converted comprises at least one image to be converted;
the acquisition module is specifically used for acquiring an initial face texture map obtained through shooting through the terminal equipment;
the generating module is also used for generating a synthetic face image according to the target face texture map and the target conversion image after generating the target face texture map according to the target region map corresponding to the K face regions respectively;
and the display module is used for displaying the synthesized face image through the terminal equipment.
Another aspect of the present application provides a computer apparatus comprising: memory, transceiver, processor, and bus system;
wherein the memory is used for storing programs;
the processor is used for executing the program in the memory, and the processor is used for executing the methods provided by the various alternative implementations in the aspects according to the instructions in the program code;
the bus system is used to connect the memory and the processor to communicate the memory and the processor.
Another aspect of the application provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the methods of the above aspects.
In another aspect of the application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions are read from the computer-readable storage medium by a processor of a computer device, and executed by the processor, cause the computer device to perform the methods provided by the various alternative implementations of the aspects described above.
From the above technical solutions, the embodiment of the present application has the following advantages:
in the embodiment of the application, a texture and normal library construction method is provided, a first original face texture map set is firstly obtained, a first sampling face texture map set is then generated according to the first original face texture map set, a first principal component analysis parameter is determined according to the first sampling face texture map set and a target face area, finally, if the construction condition of the face texture and normal library is met, the first principal component analysis parameter is used as a texture normal basis corresponding to the target face area in the face texture and normal library, and the face texture and normal library further comprises a texture normal basis corresponding to the (K-1) face area. By the method, the ultra-high definition image can be used as the original face texture map, the original face texture map is downsampled, so that the sampled face texture map with lower definition is obtained, a new face texture and normal library can be constructed based on the ultra-high definition original face texture map and the sampled face texture map corresponding to the ultra-high definition original face texture map, the face texture map with lower definition can be restored to the ultra-high definition face texture map by using the face texture and normal library, the expression of the face texture map is enhanced, and the task of the ultra-high definition face image can be realized.
Drawings
FIG. 1 is a schematic diagram of a face texture map generation system according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a process flow for constructing a face texture and normal library in an embodiment of the present application;
FIG. 3 is a schematic diagram of an embodiment of a texture and normal library construction method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of an original face texture map in an embodiment of the present application;
FIG. 5 is a schematic diagram of a face region division manner according to an embodiment of the present application;
FIG. 6 is a schematic diagram of generating a first original face texture map according to an embodiment of the present application;
FIG. 7 is a schematic diagram of generating a second original face texture map according to an embodiment of the present application;
FIG. 8 is a schematic diagram of iteratively updating a face texture and normal library in accordance with an embodiment of the present application;
FIG. 9 is a schematic diagram of an embodiment of a texture and normal map generating method according to the present application;
FIG. 10 is a schematic diagram of outputting a target face texture map based on a face texture and a normal library according to an embodiment of the present application;
FIG. 11 is a schematic view of a scene for implementing image synthesis based on a target face texture map in an embodiment of the application;
FIG. 12 is a schematic diagram of an embodiment of a texture and normal library construction apparatus according to the present application;
FIG. 13 is a schematic diagram of an embodiment of a texture and normal map generating apparatus according to the present application;
FIG. 14 is a schematic diagram of a server according to an embodiment of the present application;
fig. 15 is a schematic structural diagram of a terminal device in an embodiment of the present application.
Detailed Description
The embodiment of the application provides a texture and normal library construction method, a texture and normal map generation method and a texture and normal map generation device, which can construct a new face texture and normal library, and can restore a face texture map with lower definition into an ultra-high definition face texture map by using the face texture and normal library, so that the expression of the face texture map is enhanced, and the task of the ultra-high definition face image can be realized.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "includes" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For creating a three-dimensional face model with a sense of realism, not only three-dimensional reconstruction of a face but also reconstruction of a face texture are required, and if only three-dimensional reconstruction of a face is performed, the sense of realism is not met. Therefore, in order to make the reconstructed three-dimensional face appear more vividly, it is necessary to describe the texture of the surface when constructing the three-dimensional face model, and how to obtain the face texture map with more details becomes an important proposition. The application provides a face texture and normal library construction method, which can construct a face texture and normal library capable of outputting a target face texture map, thereby being capable of fitting an ultra-high definition face. Three examples of application target face texture maps are provided below:
1. the field of games;
the game animation roles generated based on the target face texture map have better layering sense, and the detail of the characters can be expressed as a physical fly-over. The real full face texture can increase the visual effect of the game animation roles, and the molded game animation roles can express the characters and attitudes of real users.
2. The medical field;
the three-dimensional face model generated based on the target face texture map can simulate the effect of face shaping, fine-grained image reconstruction and synthesis of the three-dimensional face model can be used for adjusting facial organs (such as noses) more accurately, and more real face effect can be simulated.
3. The field of film and television;
the simulated face model generated based on the target face texture map can generate roles in the animation, for example, the target face texture map is synthesized on a three-dimensional model corresponding to the small panda, so that the small panda with the real face expression can be seen in the animation, and a better animation effect is achieved. The artificial face model generated based on the target face texture map can also realize face replacement, for example, in a movie and television play, an actor A cannot realize a certain expression, an actor B can realize the expression, and then the target face texture map corresponding to the actor B is used as the actor A face texture map, so that facial expression replacement is realized.
Based on the above scene, a corresponding target face texture map is required to be output through a face texture and normal library, and the application provides a method for generating the texture and normal map, which is applied to a face texture map generating system shown in fig. 1. The server related by the application can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and can also be a cloud server for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDNs), basic cloud computing services such as big data and artificial intelligent platforms and the like. The terminal device may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a palm computer, a personal computer, a smart television, a smart watch, etc. The terminal device and the server may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein. The number of servers and terminal devices is not limited either.
In practical application, based on the face texture map generation system shown in fig. 1, the target face texture map can be output in the following two ways.
Firstly, generating a target face texture map on line;
the face texture and the normal library are deployed on the server side, a user can shoot or download a low-resolution face image through the terminal equipment, the terminal equipment can directly generate a corresponding initial face texture map according to the face image, the face image can also be sent to the server, and the server generates the corresponding initial face texture map according to the face image. If the initial face texture map is generated by the terminal device, the initial face texture map needs to be further transmitted to the server. The server can output a high-definition face texture map corresponding to the initial face texture map based on the face texture and the normal library.
Generating a target face texture map offline;
the face texture and normal library are deployed at the side of the terminal equipment, a user can shoot or download a low-resolution face image through the terminal equipment, the terminal equipment can directly generate a corresponding initial face texture map according to the face image, and then the terminal equipment can output a high-definition face texture map corresponding to the initial face texture map based on the face texture and normal library.
The application provides a method for constructing a face texture and normal library, the implementation flow is shown in fig. 2, please refer to fig. 2, fig. 2 is a schematic diagram of a process flow for constructing a face texture and normal library in the embodiment of the application, and the process flow is shown in the drawings, specifically:
in step S1, the data preparation stage needs to obtain an original face texture map first, and then the original face texture map may be subjected to disturbance processing to obtain more original face texture maps. The original face texture map is an ultra-high definition image and has higher resolution, so that downsampling processing is needed to be carried out on each original face texture map to obtain a corresponding sampled face texture map.
In step S2, the original face texture map and the corresponding sampled face texture map are used as samples for constructing a face texture and normal library, and the samples are used for performing iterative updating on the textures and normal vector bases in the face texture and normal library for a plurality of times until the textures and normal vector bases capable of fitting the high-definition face texture map are obtained. The texture and normal vector base can be understood as basic information of a face texture and normal library, and a great number of textures and normal patterns can be obtained through the combination of the information.
In step S3, according to the texture and the normal vector base output in step S2, the construction of the face texture and the normal library is implemented.
The method provided by the application can realize the construction of the human face texture and normal library and the generation of the human face texture map based on the artificial intelligence (Artificial Intelligence, AI) technology. The iterative training of the face texture and the normal library can be realized based on Machine Learning (ML) until the face texture and the normal library which can be put into practical application are constructed, and the generation of the face texture map can be realized based on Computer Vision (CV) technology.
Wherein artificial intelligence is the intelligence of simulating, extending and expanding a person using a digital computer or a machine controlled by a digital computer, sensing the environment, obtaining knowledge, and using knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.
The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.
The computer vision is a science for researching how to make a machine "see", and more specifically, a camera and a computer are used to replace human eyes to identify and measure targets and perform graphic processing, so that the computer is processed into images more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, and map construction, among others, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and others.
The machine learning is a multi-field interdisciplinary, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.
With research and advancement of artificial intelligence technology, research and application of artificial intelligence technology is being developed in various fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, automatic driving, unmanned aerial vehicles, robots, smart medical treatment, smart customer service, etc., and it is believed that with the development of technology, artificial intelligence technology will be applied in more fields and with increasing importance value.
In connection with the above description, the solution provided by the embodiment of the present application relates to techniques such as machine learning and computer vision of artificial intelligence, and a method for constructing a texture and normal library in the present application will be described first, referring to fig. 3, and one embodiment of the method for constructing a texture and normal library in the embodiment of the present application includes:
101. Acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map has K face areas, N is an integer greater than 1, and K is an integer greater than 1;
in this embodiment, the texture and normal library construction device acquires a first original face texture map set, where the first original face texture map set includes N first original face texture maps, each of the first original face texture maps is divided into K face regions, and each of the first original face texture maps belongs to an Ultra high-definition (Ultra HD) image. The term "ultra-high definition" refers to the formal name of the information display "4K resolution (i.e., 3840×2160 pixels)" newly approved by the international telecommunications union, and the ultra-high definition source capacity is very large. Meanwhile, "ultra-high definition" is also applicable to "8K resolution (7680×4320 pixels)", "12K resolution (11520 ×6480 pixels)", and "16K resolution (15360×8640 pixels)", etc., and the original face texture map size referred to in the present application may be 4096×4096 pixels, however this should not be construed as limiting the present application.
It will be appreciated that the texture and normal library construction device may be deployed on a server, or may be deployed on a terminal device, which is not limited herein.
102. Generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
in this embodiment, the texture and normal library construction device performs downsampling processing on each first original face texture map in the first original face texture map set to obtain corresponding first sampled face texture maps respectively. And after the downsampling processing is carried out on all the N first original face texture maps, a first sampled face texture map set can be obtained. Similar to the first original face texture map, each first sampled face texture map is also divided into K face regions. The sampled face texture map size referred to in the present application may be 512 x 512 pixels, however this should not be construed as limiting the application.
Since the high-precision first original face texture map has a very large data volume, the high-precision first original face texture map belongs to a final output result in practical application. If the first original face texture map is directly used for constructing the face texture and normal library, when the face texture and normal library is used, the initial face texture map with the same size is required to be input, however, in a real situation, the initial face texture map is unrealistic, because a consumer-level camera, such as a front lens of a mobile phone, is difficult to directly obtain the face texture map with high precision. Therefore, the application needs to output the small-size face texture map through downsampling for all the acquired high-precision original face texture maps (including the first original face texture map). For divided face regions, downsampling is also required to have the same size as a small-size face texture map.
For ease of understanding, referring to fig. 4, fig. 4 is a schematic diagram of an original face texture map in an embodiment of the present application, where, as shown in the drawing, a 4096×4096 pixel size map is taken from an entire ultra-high definition face texture map as an original face texture map, and the original face texture map may be understood as an original face texture map obtained initially, where the original face texture map is generally represented as a UV unfolded map, and U and V refer to horizontal axes and vertical axes of a 2D space.
Based on this, after the original face texture map is obtained, the original face texture map needs to be divided into K face regions, and the present application is described by taking 10 face regions as an example, but this should not be construed as limiting the present application. Referring to fig. 5, fig. 5 is a schematic diagram of a face region division manner in the embodiment of the present application, where, as shown in the drawing, a face region indicated by a mark "1" represents a face contour region, a face region indicated by a mark "2" represents a left eye region, a face region indicated by a mark "3" represents a right eye region, a face region indicated by a mark "4" represents a left eyebrow region, a face region indicated by a mark "5" represents a right eyebrow region, a face region indicated by a mark "6" represents a nose region, a face region indicated by a mark "7" represents a mouth region, a face region indicated by a mark "8" represents a chin region, a face region indicated by a mark "9" represents a left cheek region, and a face region indicated by a mark "10" represents a right cheek region.
In order to facilitate subsequent calculation, information such as coordinates of the junction of each face area can be recorded.
103. Determining a first principal component analysis parameter according to a first sampling face texture map set aiming at a target face region, wherein the target face region belongs to any face region in the K face regions;
in this embodiment, considering that the size of the first original face texture map is very large, if the entire first original face texture map is directly processed, the data processing amount is too large, resulting in lower construction efficiency of the face texture and the normal library. Therefore, in the practical training, each face region in the K face regions may be separately and iteratively trained, for convenience of explanation, any face region in the K face regions is described by taking any face region as an example, where any face region is called a "target face region", and the target face region may be a left eye region or a nose region, etc., and is not limited herein. Specifically, for the target face region, principal component analysis (Principal Component Analysis, PCA) is required for the first sampled face texture atlas to obtain a first PCA parameter.
104. And aiming at the target face region, if the face texture and normal library construction condition is met, taking the first principal component analysis parameter as a texture normal basis corresponding to the target face region in the face texture and normal library, wherein the face texture and normal library also comprises a texture normal basis corresponding to the (K-1) face region.
In this embodiment, in combination with the description of step 103, for the target face area, it is determined whether the face texture and normal library construction condition is currently satisfied, if the face texture and normal library construction condition is satisfied, the first PCA parameter is determined as the texture normal base corresponding to the target face area, and if the face texture and normal library construction condition is not satisfied, the PCA parameter is iteratively updated based on continuing until the face texture and normal library construction condition is satisfied, and then the PCA parameter generated last time is determined as the texture normal base. For example, the face texture and normal line library construction condition may have two cases, in which if the difference between the loss value obtained after the previous iteration and the loss value obtained after the current iteration is less than or equal to a preset value, the face texture and normal line library construction condition is considered to be satisfied, and if the difference between the loss value obtained after the previous iteration and the loss value obtained after the current iteration is greater than the preset value, the face texture and normal line library construction condition is considered not to be satisfied. In the second case, the number of current iterations is obtained, if the number of current iterations reaches an iteration number threshold, the face texture and normal library construction condition is considered to be satisfied, and if the number of current iterations does not reach the iteration number threshold, the face texture and normal library construction condition is considered to be not satisfied. In practical application, other ways can be adopted to determine whether the face texture and normal library construction conditions are met at present, which is not listed here.
In the process of constructing the face texture and normal library, each face area is independently processed. For convenience of explanation, the following embodiments will be described with reference to the operation for the target face region as an example. It will be appreciated that the remaining (K-1) face regions also determine the corresponding texture normal basis in a similar manner, thereby obtaining K sets of texture normal basis, and finally constructing a face texture and normal library based on the K sets of texture normal basis. The face texture and normal library provided by the application can also be called a face 3D deformable model library (3D morphable model,3DMM).
In the embodiment of the application, a texture and normal library construction method is provided, a first original face texture map set is firstly obtained, a first sampling face texture map set is then generated according to the first original face texture map set, a first principal component analysis parameter is determined according to the first sampling face texture map set and a target face area, finally, if the construction condition of the face texture and normal library is met, the first principal component analysis parameter is used as a texture normal basis corresponding to the target face area in the face texture and normal library, and the face texture and normal library further comprises a texture normal basis corresponding to the (K-1) face area. By the method, the ultra-high definition image can be used as the original face texture map, the original face texture map is downsampled, so that the sampled face texture map with lower definition is obtained, a new face texture and normal library can be constructed based on the ultra-high definition original face texture map and the sampled face texture map corresponding to the ultra-high definition original face texture map, the face texture map with lower definition can be restored to the ultra-high definition face texture map by using the face texture and normal library, the expression of the face texture map is enhanced, and the task of the ultra-high definition face image can be realized.
Optionally, based on the embodiment corresponding to fig. 3, in an optional embodiment provided by the embodiment of the present application, the obtaining a first original face texture map set includes the following steps:
acquiring a second original face texture map set, wherein the second original face texture map set comprises M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
for any one of the second original face texture maps in the second original face texture map set, each second original face texture map belongs to an ultra-high definition image. And performing disturbance processing on the second original face texture map to obtain at least one first original face texture map contained in the first original face texture map set, wherein the disturbance processing comprises at least one of rotation, translation and scaling.
In this embodiment, a manner of generating a first set of original face texture maps is described. The second original face texture map set is first required to be acquired, where the second original face texture map set is an original face texture map set used in a previous iteration, for example, the second original face texture map set is used in a 10 th iteration, and then in an 11 th iteration, the first original face texture map set is generated based on the second original face texture map set, and then the first original face texture map set is used.
Specifically, taking K equal to 10 as an example, that is, each second original face texture map is divided into 10 face regions, 4 face regions (for example, left-eye region, right-eye region, mouth region, and nose region) of the 10 face regions are randomly selected to be disturbed, that is, 2 is present 4 -1=15 choices possible, based on which at least one of rotation, translation and scaling is randomly added again, i.e. there is 2 3 -1 = 7 disturbance types. In practical applications, in order to collect more high-precision original face texture maps, more face regions may be further divided, for example, a face contour region may be divided into an upper contour region and a lower contour region, and for example, an eyebrow region may be divided into a forehead region and an eyebrow region, which is not limited herein.
It should be noted that the purpose of the data perturbation operation is to expand the expressive power of the data. Therefore, only the original face texture map is required to be operated, and the sampled face texture map is not required to be disturbed.
For convenience of explanation, referring to fig. 6, fig. 6 is a schematic diagram of generating a first original face texture map in an embodiment of the present application, where as shown in fig. 6, the (a) map is a second original face texture map, the second original face texture map is translated to obtain a first original face texture map shown in the (B) map in fig. 6, the second original face texture map is rotated to obtain a first original face texture map shown in the (C) map in fig. 6, and the second original face texture map is scaled to obtain a first original face texture map shown in the (D) map in fig. 6.
In addition, in the embodiment of the present application, a method for generating a first original face texture map set is provided, by which the fact that the process of collecting the original face texture map is complicated and the total data size is small is considered, so that a second original face texture map can be processed in a data disturbance manner, and a large number of expanded first original face texture maps are obtained, thereby improving the data size. In addition, when the disturbance processing is carried out, each face area can be disturbed independently, so that the expression capacity of each face cell is improved, and the fitting effect is further improved.
Optionally, on the basis of the embodiment corresponding to fig. 3, in another optional embodiment provided by the embodiment of the present application, the obtaining a second original face texture map set includes the following steps:
obtaining M face texture maps, wherein each face texture map comprises data of a red channel, data of a green channel and data of a blue channel;
obtaining M face normal maps, wherein each face normal map comprises data of a red channel, data of a green channel and data of a blue channel, and the face normal maps and the face texture map have a corresponding relation;
And combining each face texture map with the corresponding face normal map to obtain a second original face texture map set, wherein each second original face texture map comprises data of two red channels, data of two green channels and data of two blue channels.
In this embodiment, a manner of generating a second original face texture map based on a face texture map and a face normal map is described. Considering that the second original face texture map set includes M second original face texture maps, for convenience of description, an example of synthesizing any one second original face texture map will be described below, and the synthesis manner of the remaining (M-1) second original face texture maps is similar, which is not repeated here.
Specifically, a face texture map and a face normal map corresponding to the same face are firstly obtained, and then the face texture map and the face normal map are combined on an image channel, so that a corresponding second original face texture map is obtained. The face texture map and the face normal map are usually UV developed maps, and can be obtained from a face photo taken from multiple angles by an art engineer, or from a face three-dimensional scanning result. The face normal map describes a vector perpendicular to a tangential plane in which the current vertex is located, the face normal map has Red Green Blue (RGB) channels, and data of the RGB channels respectively represent coordinates of XYZ axes. Similarly, the face texture map also has RGB channels.
For convenience of explanation, referring to fig. 7, fig. 7 is a schematic diagram of generating a second original face texture map according to an embodiment of the present application, where as shown in fig. 7 (a) is a face texture map, fig. 7 (B) is a face normal map, and after the face texture map and channels of the face normal map are combined together, a second original face texture map with 6 channels as shown in fig. (C) is obtained.
It should be noted that, based on the foregoing embodiment, in the process of performing data perturbation on the second original face texture image, not only the face texture image included in the second original face texture image needs to be perturbed, but also the face normal image included in the second original face texture image needs to be perturbed at the same time, and the perturbation types and the perturbation positions of the two are completely consistent. It will be appreciated that the first original face texture map may also be an original face texture map having 6 channels. As described above, since the face texture map can be regarded as equivalent to the face normal map, the same zoning operation can be performed on the face normal map, that is, the face normal map also has K face regions.
In the embodiment of the application, a mode of generating the second original face texture map based on the face texture map and the face normal map is provided, by adopting the mode, in order to better express the original face texture map, the face normal map and the face texture map can be acquired by adopting high-end acquisition equipment, and one original face texture map can be expressed from 6 channels after the two are combined, so that the method has better expression effect compared with the face texture maps of 3 channels.
Optionally, on the basis of the embodiment corresponding to fig. 3, after obtaining the first original face texture atlas, another optional embodiment provided by the embodiment of the present application may further include the following steps:
aiming at a target face area, acquiring a first high mean value, a first high principal component coefficient and a first high principal component standard deviation according to a first original face texture atlas;
determining a first high principal component according to the first high principal component coefficient and the first high principal component standard deviation for the target face region;
for a target face region, determining a first principal component analysis parameter according to a first sampled face texture map set, including:
Aiming at a target face area, acquiring a first low mean value, a first low principal component coefficient and a first low principal component standard deviation according to a first sampling face texture atlas;
determining a first low principal component according to the first low principal component coefficient and the first low principal component standard deviation for the target face region;
and aiming at the target face area, acquiring a first principal component analysis parameter according to the first high average value, the first low average value, the first high principal component and the first low principal component.
In this embodiment, a way of obtaining the first PCA parameter is presented. In the following, the first PCA parameter of the target face area is taken as an example for illustration, and it can be understood that the manner of calculating the PCA parameters of other face areas is similar, so that details are not repeated here.
Specifically, a first original face texture map set (tex_al1) and a first sampling face texture map set (tex_small1) are obtained, wherein the first original face texture map set (tex_al1) comprises N first original face texture maps, and the first sampling face texture map set (tex_small1) comprises N first sampling face texture maps. Assuming that the length of each first original face texture map is W, the width of each first original face texture map is also W, and the length of each first sampling face texture map is also W, and the width of each first sampling face texture map is also W. As described in the foregoing embodiment, assuming that each first original face texture map and each first sampled face texture map have 6 channels, the first original face texture map may be represented as a column vector of [6×wx1], and similarly, the first sampled face texture map may be represented as a column vector of [6×wx1 ]. Assuming that each first original face texture map and each first sampled face texture map have 3 channels, respectively, the first original face texture map may be represented as a column vector of [3×wx1], and similarly, the first sampled face texture map may be represented as a column vector of [3×wx1 ]. This embodiment is illustrated with 6 channels as an example, which should not be construed as limiting the application.
Based on this, the first original face texture map set (tex_all1) may be represented as a two-dimensional matrix of [6×w ] ×n, and the first sampled face texture map set (tex_smal1) may be represented as a two-dimensional matrix of [6×w ] ×n. In each iteration process, firstly, the texture mean value of the first sampling face texture map set (tex_small1) in the current face region (e.g. the target face region) needs to be calculated, and then, each pixel value of the first sampling face texture map set (tex_small1) in the current face region (e.g. the target face region) is subtracted by the texture mean value. Assuming that the first sampled face texture map set (tex_small1) includes 100 first sampled face texture maps, the texture mean is the mean of the 100 first sampled face texture maps in the target face area, and since the images can be represented by using a matrix, the numerical subtraction can represent the characteristics of the target face area.
Then, PCA decomposition is performed on the target face region in the first sampled face texture map set (tex_small1), thereby obtaining a first low average value (l_mu1), a first low principal component coefficient (l_pc1), and a first low principal component standard deviation (l_ev1), wherein the first low principal component coefficient (l_pc1) is a feature vector matrix of the covariance matrix, and each column is a feature vector. The feature vectors are arranged in order from large to small, the expression capability is high, in general, PCA decomposition is to obtain a limited component which can express the original data most, for example, the original data has 1000 dimensions, and the original data can be expressed basically through the first 10 largest dimensions, so that dimension reduction is realized. In actual calculation, only the result of multiplication between the diagonal matrix formed by the first low principal component coefficient (l_pc1) and the first low principal component standard deviation (l_ev1) needs to be determined, and the first low principal component (l_pcev1) is obtained.
Similarly, PCA decomposition is performed on the target face region in the first original face texture map set (tex_all 1), so as to obtain a first high average value (h_mu1), a first high principal component coefficient (h_pc1), and a first high principal component standard deviation (h_ev1), where the first high principal component coefficient (h_pc1) is a feature vector matrix of the covariance matrix, and each column is a feature vector. The feature vectors are arranged in order from large to small, and the expression capability is high. In actual calculation, only the result of multiplication between the diagonal matrix formed by the first high principal component coefficient (h_pc1) and the first high principal component standard deviation (h_ev1) needs to be determined, and the first high principal component (h_pcev1) is obtained.
Finally, the first high mean value (h_mu1), the first low mean value (l_mu1), the first high principal component (h_pcev1) and the first low principal component (l_pcev1) are used as first principal component analysis parameters. Wherein the first low average value (l_mu1) and the first low principal component (l_pcev1) correspond to a plot of small size, and the first high average value (h_mu1) and the first high principal component (h_pcev1) correspond to a plot of original size.
In the embodiment of the application, a method for acquiring the first PCA parameter is provided, by which the required first PCA parameter can be directly acquired after PCA decomposition is respectively carried out on the first sampling face texture map set and the first original face texture map set, so that the feasibility and operability of the scheme are improved.
Optionally, on the basis of the embodiment corresponding to fig. 3, another optional embodiment provided by the embodiment of the present application determines, for a target face area, a first principal component analysis parameter according to a first sampled face texture map set, and specifically includes the following steps:
aiming at a target face area, acquiring a first low mean value, a first low principal component coefficient and a first low principal component standard deviation according to a first sampling face texture atlas;
aiming at a target face area, acquiring a first high mean value according to a first original face texture atlas;
acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with a first original face texture map;
for a target face region, determining a first high principal component according to the first low mean value, the first low principal component coefficient, the first low principal component standard deviation, the first high mean value and the first texture fitting coefficient;
and aiming at the target face area, acquiring a first principal component analysis parameter according to the first high average value, the first low average value, the first high principal component and the first low principal component.
In this embodiment, another way of obtaining the first PCA parameter is presented. In the following, the first PCA parameter of the target face area is taken as an example for illustration, and it can be understood that the manner of calculating the PCA parameters of other face areas is similar, so that details are not repeated here.
Similar to the previous embodiment, the first original face texture map set (tex_all1) may be represented as a two-dimensional matrix of [6×w ] ×n, and the first sampled face texture map set (tex_smal1) may be represented as a two-dimensional matrix of [6×w ] ×n. In each iteration process, firstly, the texture mean value of the first sampling face texture map set (tex_small1) in the current face region (e.g. the target face region) needs to be calculated, and then, each pixel value of the first sampling face texture map set (tex_small1) in the current face region (e.g. the target face region) is subtracted by the texture mean value.
Specifically, PCA decomposition is performed on a target face region in the first sampled face texture map set (tex_small1), so as to obtain a first low average value (l_mu1), a first low principal component coefficient (l_pc1), and a first low principal component standard deviation (l_ev1), where the first low principal component coefficient (l_pc1) is a feature vector matrix of the covariance matrix, and each column is a feature vector. The feature vectors are arranged in order from large to small, and the feature vectors are placed in front of the feature vectors with high expression capacity, so that dimension reduction is realized. In actual calculation, only the result of multiplication between the diagonal matrix formed by the first low principal component coefficient (l_pc1) and the first low principal component standard deviation (l_ev1) needs to be determined, and the first low principal component (l_pcev1) is obtained. The use of the first low principal component (l_pcev1) and the first texture fitting coefficients enables the recovery of the complete face texture map and face normal map, and therefore the subsequent use of the first low principal component (l_pcev1) is sufficient.
The PCA decomposition is performed for the target face region in the first original face texture map set (tex_all 1), and since the number of first original face texture maps is very large, the process of calculating the first high principal component coefficient (h_pc1) and the first high principal component standard deviation (h_ev1) is also very complicated. However, since each column in the first sampled face texture map set (tex_small1) represents the same data as each column in the first original face texture map set (tex_small1) only a difference in size exists, the face texture map may be fitted with a first low mean value (l_mu1), a first low principal component coefficient (l_pc1), and a first low principal component standard deviation (l_ev1), which is represented as a diagonal matrix. After the first texture fitting coefficients are obtained, the first texture fitting coefficients should be equally applicable to the generation of the original size map. Based on this, the first high principal component coefficient (h_pc1) and the first high principal component standard deviation (h_ev1) will be calculated as follows:
tex_small (part) =l_mu (part) +l_pc (part) ×l_ev (part) ×id; (1)
tex_all (part) =h_mu (part) +h_pc (part) ×h_ev (part) ×id; (2)
Wherein part represents a target face region, tex_small represents a sampled face texture map set, tex_all represents an original face texture map set, l_mu represents a low mean value, l_pc represents a low principal component coefficient, l_ev represents a low principal component standard deviation, h_mu represents a high mean value, h_pc represents a high principal component coefficient, and h_ev represents a high principal component standard deviation. It is noted that the low principal component standard deviation (l_ev) and the high principal component standard deviation (h_ev) are both diagonal matrices. id represents a texture fitting coefficient, and is represented as N texture vectors, where each texture vector in the N texture vectors has a corresponding relationship with the first original face texture map, and may be specifically represented as a feature matrix of N columns and T rows.
In equations 1 and 2, only h_pc and h_ev are unknowns, and the others are known, so that the results of h_pc and h_ev can be obtained by combining equations 1 and 2. That is, the values of h_pc and h_ev are not required to be known, and the complete face texture map and the face normal map can be recovered according to the texture fitting coefficients only by knowing the results of h_pc. Thus, h_pc and h_ev are not subdivided below, pc×ev being collectively referred to as pcev.
It should be noted that, before tex_all (part) calculates h_pc_h_e, a mean removal operation is also required to be performed, that is, a texture mean value of the original face texture map set (tex_all) in the current face area (e.g., the target face area) is calculated, and then each pixel value of the original face texture map set (tex_all) in the current face area (e.g., the target face area) is subtracted by the texture mean value. It should be further noted that tex_small (part) is texture information of a current face region (e.g., a target face region), and does not include normal information, and tex_small may include normal information or may not include normal information.
In summary, the first high average value (h_mu1), the first low average value (l_mu1), the first high principal component (h_pcev1), and the first low principal component (l_pcev1) are used as the first principal component analysis parameters. Wherein the first low average value (l_mu1) and the first low principal component (l_pcev1) correspond to a plot of small size, and the first high average value (h_mu1) and the first high principal component (h_pcev1) correspond to a plot of original size.
In the embodiment of the application, another mode of acquiring the first PCA parameter is provided, by which after the PCA decomposition is performed on the first sampled face texture atlas set, the first high principal component can be directly deduced based on the first low mean value, the first low principal component coefficient and the first low principal component standard deviation corresponding to the first sampled face texture atlas set, so that the problem of excessive calculation amount caused by directly performing the PCA decomposition on the target face area in the first original face texture atlas set is avoided, and the feasibility and convenience of the scheme are improved.
Optionally, based on the embodiment corresponding to fig. 3, another optional embodiment provided by the present application specifically includes the following steps, for a target face area, of acquiring a first low average value, a first low principal component coefficient, and a first low principal component standard deviation according to a first sampled face texture atlas:
aiming at a target face area, carrying out principal component analysis on a first sampling face texture atlas to obtain a first low-average value and a low principal component coefficient to be processed, wherein the low principal component coefficient to be processed comprises Q feature vectors, and Q is an integer larger than 1;
Aiming at a target face area, acquiring a low principal component variance to be processed according to the low principal component coefficient to be processed, wherein the low principal component variance to be processed comprises Q characteristic values, and the characteristic values and the characteristic vectors have a corresponding relation;
aiming at a target face area, arranging Q characteristic values included in the variance of the low principal component to be processed according to the sequence from big to small to obtain Q characteristic values after sequencing;
for a target face region, T characteristic values with the characteristic value duty ratio larger than a duty ratio threshold value are obtained from the Q characteristic values after sequencing, wherein T is an integer larger than or equal to 1 and smaller than or equal to Q;
aiming at a target face area, acquiring a first low principal component standard deviation according to T characteristic values;
for a target face region, determining T feature vectors corresponding to the T feature values from Q feature vectors included in the low principal component coefficients to be processed, and acquiring a first low principal component coefficient according to the T feature vectors.
In this embodiment, a manner of acquiring the first low principal component standard deviation based on PCA decomposition is described. In the following, the first low average value, the first low principal component coefficient and the first low principal component standard deviation of the target face area are calculated as examples, and it is understood that the way of calculating the low average value, the low principal component coefficient and the low principal component standard deviation of other face areas is similar, so that details are not repeated here.
Specifically, for any face region (for example, a target face region), PCA decomposition is first performed on a first sampled face texture map set (tex_small1) to obtain a first low average value (l_mu1) and a low principal component coefficient to be processed, where the low principal component coefficient to be processed includes Q feature vectors. And acquiring a low principal component variance to be processed according to the low principal component coefficient to be processed, wherein the low principal component variance to be processed is a characteristic value of the low principal component coefficient to be processed, and therefore, the low principal component variance to be processed comprises Q characteristic values, and the characteristic values in the Q characteristic values and the characteristic vectors of the Q characteristic vectors have a one-to-one correspondence. Based on this, Q feature values can be ranked from large to small, and the larger the feature value, the more the feature value can express the feature. Assuming that Q is 5, the Q feature values after sorting are [1,0.8,0.5,0.3,0.1] respectively, so that T feature values with a duty ratio greater than a duty ratio threshold value can be selected from the Q feature values after sorting, and assuming that the duty ratio threshold value is 0.9, then the duty ratio relationships including different feature values are calculated respectively.
Taking the maximum eigenvalue as an example, the duty ratio is 1/(1+0.8+0.5+0.3+0.1) =0.37, and since 0.37 is smaller than 0.9, it is necessary to continue to take the next eigenvalue for calculation.
Then, the duty ratio of the first two eigenvalues is calculated, and the duty ratio is (1+0.8)/(1+0.8+0.5+0.3+0.1) =0.67, and since 0.67 is smaller than 0.9, it is necessary to continue to take the next eigenvalue for calculation.
Then, the duty ratio of the first three eigenvalues is calculated to be (1+0.8+0.5)/(1+0.8+0.5+0.3+0.1) =0.85, and since 0.85 is smaller than 0.9, it is necessary to continue the calculation with the next eigenvalue.
Then, the duty ratio of the first four eigenvalues is calculated, which is (1+0.8+0.5+0.3)/(1+0.8+0.5+0.3+0.1) =0.96, and since 0.96 is larger than 0.9, it is determined to take out the first 4 eigenvalues as the eigenvalues of the subsequent calculation, which is exemplified by t=4.
After determining the T eigenvalues, the first low principal component standard deviation may be obtained according to the T eigenvalues, and at the same time, the eigenvectors corresponding to the T eigenvalues are respectively used as the first low principal component coefficients (l_pc1) according to the correspondence between the eigenvalues and the eigenvectors. Based on this, taking the 6-channel image as an example, it can be determined that the first low mean value (l_mu1) is represented as a vector of [6×w×wx1], the first low principal component coefficient (l_pc1) is represented as a matrix of [6×w×wxt ], and the first low principal component variance (l_ev_f1) is represented as a vector of [ t×1 ]. And then the first low principal component variance (l_ev_f1) is changed into a diagonal matrix after the root number is opened, so that the first low principal component standard deviation (l_ev1) is obtained, and the first low principal component standard deviation (l_ev1) is expressed as a diagonal matrix of [ T ] T.
In the embodiment of the application, a mode of acquiring the standard deviation of the first low principal component based on PCA decomposition is provided, and the dimension reduction of the data can be realized based on PCA decomposition by the mode, so that the difficulty of data processing is reduced, and the construction efficiency of the face texture and the normal library is further improved.
Optionally, on the basis of the embodiment corresponding to fig. 3, another optional embodiment provided by the embodiment of the present application further includes, for the target face area, after determining the first principal component analysis parameter according to the first sampled face texture map set, the steps of:
acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with a first original face texture map;
for a target face area, determining N first loss values according to a first texture fitting coefficient, a first original face texture map set and a first high mean value and a first high principal component included in a first principal component analysis parameter, wherein the first loss values and the first original face texture map have a corresponding relation;
for a target face region, if the difference value between the maximum loss value in the N first loss values and the maximum loss value in the M second loss values is smaller than or equal to a difference threshold value, determining that the face texture and normal library construction condition is met, wherein the M second loss values are determined according to a second original face texture image set, and the second original face texture image set is subjected to disturbance processing to generate a first original face texture image set.
In this embodiment, a manner of calculating the loss value based on the first PCA parameter is described, and the following will take N first loss values of the calculation target face area as an example, and it will be understood that the loss value calculation manners of other face areas are similar, so that no description is given here.
Specifically, in the current iteration, for the first original face texture map set (tex_all 1), N first original face texture maps are in total, so the first original face texture map set (tex_all 1) may be represented as a two-dimensional matrix of [6×w×w ] ×n, and by fitting the texture portion of the target face region, the first texture fitting coefficient may be obtained as [ t×n ], where the value manner of T is described in the foregoing embodiment, which is not described herein. Based on this, the loss value can be calculated as follows:
loss＝Σ x∈part (tex_dis (x) - (h_mu+h_pcev) (x)); (3)
Where part represents a target face region, tex_dis represents an original face texture map set (for example, a first original face texture map set (tex_all1)) obtained after disturbance, h_mu represents a high average value, h_pc represents a high principal component coefficient, and h_ev represents a high principal component coefficient. id denotes a texture fitting coefficient, and x denotes a pixel position in the target face region. tex_dis (x) represents the xth data in each column. (h_mu+h_pcev) id (x) represents the xth data of each column after the small size is restored to the original size, and the data of each row is accumulated to obtain a row vector of [1*N ].
For example, based on equation 3, after substituting the first texture fitting coefficient, the first original face texture map set (tex_all 1), the first high average value (h_mu1), and the first high principal component (h_pcev1), a row vector of [1*N ] may be calculated, and N first loss values may be obtained.
Since the second principal component analysis parameter obtained in the previous iteration has been obtained in the current iteration, M second loss values may also be obtained based on the second principal component analysis parameter, where M second loss values are represented as a row vector of [1*M ]. Thus, a maximum loss value may be determined from the N first loss values, and a maximum loss value may be determined from the M second loss values, where if the difference between the two maximum loss values is less than or equal to the difference threshold, the iteration is considered to have converged, i.e. the face texture and normal library construction condition is satisfied. The difference threshold may be 0.1, or 0.01, etc., which is not limited herein.
In the embodiment of the application, a mode of calculating the loss value based on the first PCA parameter is provided, by the mode, the loss value is compared between the second original face texture image before data disturbance and the first original face texture image set obtained after data disturbance, and the compared loss values are the maximum values in the loss values respectively, so that the difference condition can be expressed better, and the robustness of the face texture and the normal library is enhanced.
Optionally, on the basis of the embodiment corresponding to fig. 3, before obtaining the first original face texture atlas, another optional embodiment provided in the embodiment of the present application further includes the following steps:
acquiring a second original face texture map set, wherein the second original face texture map set comprises M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
generating a second sampling face texture map set according to the second original face texture map set, wherein the second sampling face texture map set comprises M second sampling face texture maps, each second sampling face texture map is obtained by downsampling the second original face texture map, and each second sampling face texture map is provided with K face areas;
determining a second principal component analysis parameter according to a second sampling face texture map set aiming at the target face region, wherein the second principal component analysis parameter comprises a second high average value, a second low average value, a second high principal component and a second low principal component;
obtaining a second texture fitting coefficient, wherein the second texture fitting coefficient comprises M texture vectors, and each texture vector in the M texture vectors has a corresponding relation with a second original face texture map;
And aiming at the target face area, determining M second loss values according to a second texture fitting coefficient, a second original face texture map set and a second high mean value and a second high principal component included by a second principal component analysis parameter, wherein the second loss values and the second original face texture map have a corresponding relation.
In this embodiment, a manner of obtaining the loss value in the previous iteration process is described, and an example of calculating M second loss values of the target face area will be described below, where it can be understood that the loss value calculation manners of other face areas are similar, and therefore will not be described herein.
Specifically, the second original face texture map set (tex_all2) is an original face texture map set used in the previous iteration, and a corresponding second sampled face texture map set (tex_smal2) can be obtained according to the second original face texture map set (tex_all2), where the second original face texture map set (tex_all2) includes M second original face texture maps, and the second sampled face texture map set (tex_smal2) includes M second sampled face texture maps. Assuming that the length of each second original face texture map is W, the width of each second original face texture map is also W, and the length of each second sampling face texture map is also W. As described in the foregoing embodiment, assuming that each second original face texture map and each second sampled face texture map have 6 channels, the second original face texture map may be represented as a column vector of [6×wx1], and similarly, the second sampled face texture map may be represented as a column vector of [6×wx1 ]. Assuming that each second original face texture map and each second sampled face texture map have 3 channels, respectively, the second original face texture map may be represented as a column vector of [3×wx1], and similarly, the second sampled face texture map may be represented as a column vector of [3×wx1 ]. This embodiment is illustrated with 6 channels as an example, which should not be construed as limiting the application.
Based on this, the second original face texture map set (tex_all2) may be represented as a two-dimensional matrix of [6×w ] ×n, and the second sampled face texture map set (tex_smal2) may be represented as a two-dimensional matrix of [6×w ] ×n. In each iteration process, firstly, the texture mean value of the second sampled face texture map set (tex_small2) in the current face region (e.g. the target face region) needs to be calculated, and then, each pixel value of the second sampled face texture map set (tex_small2) in the current face region (e.g. the target face region) is subtracted by the texture mean value.
Then, PCA decomposition is performed on the target face region in the second sampled face texture map set (tex_small2), thereby obtaining a second low average value (l_mu2), a second low principal component coefficient (l_pc2), and a second low principal component standard deviation (l_ev2), wherein the second low principal component coefficient (l_pc2) is a feature vector matrix of the covariance matrix, and each column is a feature vector. The feature vectors are arranged in order from large to small, and the expression capability is high. Only the second low principal component coefficient (l_pc2) is determined, and the second low principal component standard deviation (l_ev2) is obtained.
Since each column in the second sampled face texture map set (tex_small2) and the second original face texture map set (tex_small2) represent the same data, only the size difference exists, so that the face texture map can be fitted by using the second low average value (l_mu2), the second low principal component coefficient (l_pc2) and the second low principal component standard deviation (l_ev2), and finally the second high principal component coefficient (h_pc2) and the second high principal component standard deviation (h_ev2) are obtained.
In summary, the second Gao Junzhi (h_mu2), the second low average value (l_mu2), the second high principal component (h_pcev2), and the second low principal component (l_pcev2) are taken as the second principal component analysis parameters. Wherein the second low average value (l_mu2) and the second low principal component (l_pcev2) correspond to a small-sized plot, and the second Gao Junzhi (h_mu2) and the second high principal component (h_pcev2) correspond to an original-sized plot.
Using equation 3 above, and determining M second loss values according to the second texture fitting coefficient, the second original face texture map set (tex_all2), the second Gao Junzhi (h_mu2), and the second high principal component (h_pcev2).
In the embodiment of the application, a method for acquiring the loss value in the previous iteration process is provided, by the method, the comparison of the loss values between two iterations can be realized, and each loss value has a corresponding relation with the original face texture map, so that the difference between the original face texture maps can be expressed more accurately.
Optionally, on the basis of the embodiment corresponding to fig. 3, in another optional embodiment provided by the embodiment of the present application, the method further includes:
aiming at a target face region, if the difference value between the maximum loss value in the N first loss values and the maximum loss value in the M second loss values is larger than a difference value threshold value, arranging the N first loss values in the order from large to small to obtain N first loss values after sequencing;
for a target face region, acquiring the first loss values of the first N numbers of the first loss values which are arranged in front from the first loss values of the N numbers of the first loss values which are arranged in sequence, wherein H is an integer which is larger than or equal to 1 and smaller than or equal to N;
determining corresponding H first original face texture maps according to the H first loss values;
and carrying out disturbance processing on the H first original face texture images to obtain at least one third original face texture image, wherein the disturbance processing comprises at least one of rotation, translation and scale scaling.
In this embodiment, a manner of iteratively updating the face texture and the normal library is described, and processing of the target face area will be described as an example, and it can be understood that the processing manner of other face areas is similar, so that details are not repeated here.
For easy understanding, referring to fig. 8, fig. 8 is a schematic diagram of iteratively updating a face texture and normal library according to an embodiment of the present application, where an original face texture map set is acquired before the face texture and normal library is constructed, where the original face texture map included in the set may be captured by a sampling device. Because the difficulty of sampling the original face texture images is considered to be large, the original face texture images can be subjected to disturbance processing, so that more original face texture images are obtained, and corresponding loss values are calculated based on the disturbed original face texture image sets. Assume that in the 300 th iteration, the texture and normal basis of each face region are updated using a first set of original face texture maps, where the first set of original face texture maps includes N first original face texture maps, and N is assumed to be 200. If the difference between the maximum loss value of the N first loss values and the maximum loss value of the M second loss values is larger than the difference threshold, namely that the face texture and the normal library with good fitting effect cannot be obtained in the iteration, the maximum H first loss values are needed to be obtained from the N first loss values, and a first original face texture map corresponding to the H first loss values is used as a basis of the next iteration. That is, in the next iteration, the data perturbation is performed on the H first original face texture maps, so as to obtain at least one third original face texture map, for example, 200 third original face texture maps.
It should be noted that H may be a constant value, for example, 50 or other values. H may also be a proportion, for example, 5% of the total number, without limitation.
In the embodiment of the application, a mode of iteratively updating the face texture and the normal library is provided, by the mode, under the condition that the construction conditions of the face texture and the normal library are not met, part of original face texture images can be taken out, and disturbance processing is carried out on the original face texture images, so that more original face texture images for subsequent iteration processing are obtained. Therefore, on one hand, iteration operation can be completed without collecting a large number of original face texture images, so that time cost and financial cost are saved, and on the other hand, only the original face texture images with larger loss values are taken out for each iteration to serve as objects of subsequent fitting, and better training effect can be achieved.
With reference to the foregoing description, the method for generating a texture and normal map according to the present application will be described with reference to fig. 9, and an embodiment of the method for generating a texture and normal map according to an embodiment of the present application includes:
201. acquiring an initial face texture map, wherein the initial face texture map corresponds to a first size;
In this embodiment, the texture and normal map generating device first obtains an initial face texture map, where the initial face texture map corresponds to a first size, for example, the first size may be 512×512 pixels.
Note that, the texture and normal map generating device may be disposed in the terminal device or may be disposed in the server, which is not limited herein.
202. Determining a K face area according to the initial face texture map, wherein K is an integer greater than 1;
in this embodiment, the texture and normal map generating device divides the initial face texture map into K face regions, where the K face regions are predefined, for example, including a left eye region and a chin region.
203. And aiming at each face region included in the initial face texture map, acquiring target region maps corresponding to the K face regions respectively through the face textures and a normal library, wherein the target region maps are ultrahigh-definition region maps. The human face texture and normal library comprises K groups of texture normal bases, each group of texture normal bases corresponds to a human face region, and the human face texture and normal library is obtained by adopting the texture and normal library construction method provided by the application;
in this embodiment, the texture and normal map generating device generates a corresponding target area map for each face area, that is, a texture normal base corresponding to each face area stored in the face texture and normal library. For easy understanding, referring to fig. 10, fig. 10 is a schematic diagram of outputting a target face texture map based on a face texture and normal library according to an embodiment of the present application, where, taking K equal to 10 as an example, an initial face texture map is first divided into 10 face regions, each face region corresponds to a set of texture normal bases, and each set of texture normal bases includes 4 parameters, which are respectively a low average value (l_mu), a low principal component (l_pcev), a high average value (h_mu), and a high principal component (h_pcev). Referring to table 1, table 1 is an illustration of 10 sets of texture normals contained in a human face texture and normals library.
TABLE 1
| Face region | l_mu | l_pcev | h_mu | h_pcev |
| Face contour region | l_mu_face | l_pcev_face | h_mu_face | h_pcev_face |
| Left eye region | l_mu_leye | l_pcev_leye | h_mu_leye | h_pcev_leye |
| Right eye region | l_mu_reye | l_pcev_reye | h_mu_reye | h_pcev_reye |
| Left eyebrow areaDomain | l_mu_leyebrow | l_pcev_leyebrow | h_mu_leyebrow | h_pcev_leyebrow |
| Right eyebrow area | l_mu_reyebrow | l_pcev_reyebrow | h_mu_reyebrow | h_pcev_reyebrow |
| Nose region | l_mu_nose | l_pcev_nose | h_mu_nose | h_pcev_nose |
| Chin area | l_mu_chin | l_pcev_chin | h_mu_chin | h_pcev_chin |
| Left cheek region | l_mu_lcheek | l_pcev_lcheek | h_mu_lcheek | h_pcev_lcheek |
| Right cheek region | l_mu_rcheek | l_pcev_rcheek | h_mu_rcheek | h_pcev_rcheek |
As can be seen from table 1 and fig. 10, when using the texture normal basis, it is necessary to generate K target region maps by independently restoring each face region.
204. And generating a target face texture map according to the target region maps respectively corresponding to the K face regions, wherein the target face texture map corresponds to a second size which is larger than the first size.
In this embodiment, the texture and normal map generating device splices the target region maps corresponding to the generated K face regions, and finally achieves the purpose of restoring the low-resolution initial face texture map to the target face texture map. The target face texture map is the ultra-high definition face texture map. The initial face texture map corresponds to a second size, which may be 4096 x 4096 pixels, for example.
In the embodiment of the application, the method for generating the texture and normal map is provided, and by adopting the method, the constructed face texture and normal library can be utilized to restore the initial face texture map into the target face texture map, so that the expression of the face texture map is greatly improved, and the task of ultra-high definition face images can be realized.
Optionally, on the basis of the embodiment corresponding to fig. 9, an optional embodiment provided by the embodiment of the present application may further include the following steps:
acquiring an image selection instruction through a terminal device, wherein the image selection instruction is used for indicating a target conversion image;
acquiring a target conversion image from an image set to be converted according to an image selection instruction, wherein the image set to be converted comprises at least one image to be converted;
the method for acquiring the initial face texture map specifically comprises the following steps:
acquiring an initial face texture map obtained through shooting through terminal equipment;
after generating the target face texture map according to the target region maps respectively corresponding to the K face regions, the method may further include the following steps:
generating a synthetic face image according to the target face texture map and the target conversion image;
and displaying the synthesized face image through the terminal equipment.
In this embodiment, an application manner of generating a target face texture map based on an initial face texture map is described, and in practical application, a more realistic virtual object may be generated based on the target face texture map.
For convenience of description, referring to fig. 11, fig. 11 is a schematic view of a scene for implementing image synthesis based on a target face texture map in an embodiment of the present application, as shown in the drawing, a user may first shoot an own avatar through a terminal device (e.g. a mobile phone), and select a trigger instruction for converting an image to a target, for example, a trigger instruction for "character E", on an application interface. If the face photo shot at this time and the target conversion image are confirmed to be synthesized, clicking the confirmation, thereby generating an initial face texture map corresponding to the face photo. Based on the face texture and the texture normal basis in the normal library, target area diagrams corresponding to K face areas respectively can be obtained, and the K target area diagrams can be combined into a target face texture diagram. The target face texture map may then be composited with a target transformed image (e.g., an image of character E) to ultimately yield a rendered face image. After the terminal device acquires the synthesized face image, the synthesized face image can be displayed.
In addition, in the embodiment of the application, an application mode for generating the target face texture map based on the initial face texture map is provided, and the application mode can be applied to virtual persons and peripheral derivative products, and the low-quality initial face texture map can be converted into the high-quality target face texture map by using the face texture and a normal library, so that the application mode has very important effect on rendering super-realistic faces, for example, a game character which is the same as a user face can be constructed and own face texture can be added.
Referring to fig. 12, fig. 12 is a schematic diagram illustrating an embodiment of a texture and normal library construction device according to an embodiment of the present application, the texture and normal library construction device 30 includes:
the obtaining module 301 is configured to obtain a first original face texture map set, where the first original face texture map set includes N first original face texture maps, each first original face texture map has K face regions, N is an integer greater than 1, and K is an integer greater than 1;
the generating module 302 is configured to generate a first sampled face texture map set according to the first original face texture map set, where the first sampled face texture map set includes N first sampled face texture maps, each first sampled face texture map is obtained by downsampling the first original face texture map, and each first sampled face texture map has K face regions;
A determining module 303, configured to determine, according to the first sampled face texture map set, a first principal component analysis parameter for a target face area, where the target face area belongs to any one of the K face areas;
the construction module 304 is configured to, for a target face region, take a first principal component analysis parameter as a texture normal basis corresponding to the target face region in the face texture and normal library if a face texture and normal library construction condition is satisfied, where the face texture and normal library further includes a texture normal basis corresponding to the (K-1) face region.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the obtaining module 301 is specifically configured to obtain a second original face texture map set, where the second original face texture map set includes M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
and performing disturbance processing on any one of the second original face texture maps in the second original face texture map set to obtain at least one first original face texture map contained in the first original face texture map set, wherein the disturbance processing comprises at least one of rotation, translation and scaling.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the acquiring module 301 is specifically configured to acquire M face texture maps, where each face texture map includes data of a red channel, data of a green channel, and data of a blue channel;
obtaining M face normal maps, wherein each face normal map comprises data of a red channel, data of a green channel and data of a blue channel, and the face normal maps and the face texture map have a corresponding relation;
and combining each face texture map with the corresponding face normal map to obtain a second original face texture map set, wherein each second original face texture map comprises data of two red channels, data of two green channels and data of two blue channels.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the obtaining module 301 is further configured to obtain, for the target face area, a first high average value, a first high principal component coefficient, and a first high principal component standard deviation according to the first original face texture map set after obtaining the first original face texture map set;
The determining module 303 is further configured to determine, for the target face area, a first high principal component according to the first high principal component coefficient and the first high principal component standard deviation;
the determining module 303 is specifically configured to obtain, for a target face area, a first low average value, a first low principal component coefficient, and a first low principal component standard deviation according to a first sampled face texture atlas;
determining a first low principal component according to the first low principal component coefficient and the first low principal component standard deviation for the target face region;
and aiming at the target face area, acquiring a first principal component analysis parameter according to the first high average value, the first low average value, the first high principal component and the first low principal component.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the determining module 303 is specifically configured to obtain, for a target face area, a first low average value, a first low principal component coefficient, and a first low principal component standard deviation according to a first sampled face texture atlas;
aiming at a target face area, acquiring a first high mean value according to a first original face texture atlas;
Acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with a first original face texture map;
for a target face region, determining a first high principal component according to the first low mean value, the first low principal component coefficient, the first low principal component standard deviation, the first high mean value and the first texture fitting coefficient;
and aiming at the target face area, acquiring a first principal component analysis parameter according to the first high average value, the first low average value, the first high principal component and the first low principal component.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the determining module 303 is specifically configured to perform principal component analysis on the first sampled face texture atlas with respect to the target face area, to obtain a first low average value and a low principal component coefficient to be processed, where the low principal component coefficient to be processed includes Q feature vectors, and Q is an integer greater than 1;
aiming at a target face area, acquiring a low principal component variance to be processed according to the low principal component coefficient to be processed, wherein the low principal component variance to be processed comprises Q characteristic values, and the characteristic values and the characteristic vectors have a corresponding relation;
Aiming at a target face area, arranging Q characteristic values included in the variance of the low principal component to be processed according to the sequence from big to small to obtain Q characteristic values after sequencing;
for a target face region, T characteristic values with the characteristic value duty ratio larger than a duty ratio threshold value are obtained from the Q characteristic values after sequencing, wherein T is an integer larger than or equal to 1 and smaller than or equal to Q;
aiming at a target face area, acquiring a first low principal component standard deviation according to T characteristic values;
for a target face region, determining T feature vectors corresponding to the T feature values from Q feature vectors included in the low principal component coefficients to be processed, and acquiring a first low principal component coefficient according to the T feature vectors.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the obtaining module 301 is further configured to obtain a first texture fitting coefficient after the determining module 303 determines, for the target face area, a first principal component analysis parameter according to the first sampled face texture map set, where the first texture fitting coefficient includes N texture vectors, and each texture vector in the N texture vectors has a corresponding relationship with the first original face texture map;
The determining module 303 is further configured to determine, for the target face area, N first loss values according to the first texture fitting coefficient, the first original face texture map set, and the first high average value and the first high principal component included in the first principal component analysis parameter, where the first loss values have a corresponding relationship with the first original face texture map;
the determining module 303 is further configured to determine that the face texture and normal library construction condition is met if, for the target face region, a difference between a maximum loss value of the N first loss values and a maximum loss value of the M second loss values is less than or equal to a difference threshold, where the M second loss values are determined according to a second original face texture atlas, and the second original face texture atlas generates a first original face texture atlas after being subjected to disturbance processing.
Alternatively, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application,
the obtaining module 301 is further configured to obtain a second original face texture map set before the first original face texture map set is obtained, where the second original face texture map set includes M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
The generating module 302 is further configured to generate a second sampled face texture map set according to the second original face texture map set, where the second sampled face texture map set includes M second sampled face texture maps, each second sampled face texture map is obtained by downsampling the second original face texture map, and each second sampled face texture map has K face areas;
the determining module 303 is further configured to determine, for the target face area, a second principal component analysis parameter according to the second sampled face texture map set, where the second principal component analysis parameter includes a second high average value, a second low average value, a second high principal component, and a second low principal component;
the obtaining module 301 is further configured to obtain a second texture fitting coefficient, where the second texture fitting coefficient includes M texture vectors, and each texture vector of the M texture vectors has a corresponding relationship with the second original face texture map;
the determining module 303 is further configured to determine, for the target face area, M second loss values according to the second texture fitting coefficient, the second set of original face texture maps, and a second high average value and a second high principal component included in the second principal component analysis parameter, where the second loss values have a correspondence with the second original face texture maps.
Optionally, based on the embodiment corresponding to fig. 12, in another embodiment of the texture and normal library construction device 30 provided in the embodiment of the present application, the texture and normal library construction device 30 further includes a sorting module 305 and a perturbation module 306;
the sorting module 305 is configured to, for a target face region, sort the N first loss values according to a sequence from large to small if a difference value between a maximum loss value of the N first loss values and a maximum loss value of the M second loss values is greater than a difference threshold, so as to obtain N sorted first loss values;
the obtaining module 301 is further configured to obtain, for a target face area, H first loss values arranged in front from the N first loss values after sorting, where H is an integer greater than or equal to 1 and less than or equal to N;
the determining module 303 is further configured to determine corresponding H first original face texture maps according to the H first loss values;
the perturbation module 306 is configured to perform perturbation processing on the H first original face texture maps to obtain at least one third original face texture map, where the perturbation processing includes at least one of rotation, translation, and scaling.
Referring to fig. 13, fig. 13 is a schematic diagram illustrating an embodiment of a texture and normal map generating apparatus according to an embodiment of the present application, the texture and normal map generating apparatus 40 includes:
An obtaining module 401, configured to obtain an initial face texture map, where the initial face texture map corresponds to a first size;
a determining module 402, configured to determine K face regions according to the initial face texture map, where K is an integer greater than 1;
the obtaining module 401 is further configured to obtain, for each face region included in the initial face texture map, a target region map corresponding to each of the K face regions through a face texture and normal library, where the face texture and normal library includes K groups of texture normal bases, and each group of texture normal bases corresponds to one face region;
the generating module 403 is configured to generate a target face texture map according to the target region maps corresponding to the K face regions, where the target face texture map corresponds to a second size, and the second size is greater than the first size.
Optionally, on the basis of the embodiment corresponding to fig. 13, in another embodiment of the texture and normal map generating apparatus 40 provided in the embodiment of the present application, the texture and normal map generating apparatus 40 further includes a display module 404;
the acquiring module 401 is further configured to acquire an image selection instruction through the terminal device, where the image selection instruction is used to instruct the target to convert the image;
The obtaining module 401 is further configured to obtain, according to an image selection instruction, a target conversion image from a to-be-converted image set, where the to-be-converted image set includes at least one to-be-converted image;
the acquiring module 401 is specifically configured to acquire an initial face texture map obtained by shooting through a terminal device;
the generating module 403 is further configured to generate a synthetic face image according to the target face texture map and the target conversion image after generating the target face texture map according to the target region maps corresponding to the K face regions respectively;
and the display module 404 is used for displaying the synthesized face image through the terminal equipment.
Fig. 14 is a schematic diagram of a server structure provided by an embodiment of the present application, where the server 500 may vary considerably in configuration or performance, and may include one or more central processing units (central processing units, CPU) 522 (e.g., one or more processors) and memory 532, one or more storage media 530 (e.g., one or more mass storage devices) storing applications 542 or data 544. Wherein memory 532 and storage medium 530 may be transitory or persistent. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations on a server. Still further, the central processor 522 may be configured to communicate with a storage medium 530 and execute a series of instruction operations in the storage medium 530 on the server 500.
The Server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input/output interfaces 558, and/or one or more operating systems 541, such as Windows Server TM ，Mac OS X TM ，Unix TM ,Linux TM ，FreeBSD TM Etc.
The steps performed by the server in the above embodiments may be based on the server structure shown in fig. 14.
The embodiment of the present application further provides another generation of a face texture map, as shown in fig. 15, for convenience of explanation, only the portion relevant to the embodiment of the present application is shown, and specific technical details are not disclosed, please refer to the method portion of the embodiment of the present application. The terminal device may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), a Point of Sales (POS), a vehicle-mounted computer, and the like, taking the terminal device as an example of the mobile phone:
fig. 15 is a block diagram showing a part of the structure of a mobile phone related to a terminal device provided by an embodiment of the present application. Referring to fig. 15, the mobile phone includes: radio Frequency (RF) circuitry 610, memory 620, input unit 630, display unit 640, sensor 650, audio circuitry 660, wireless fidelity (wireless fidelity, wiFi) module 670, processor 680, power supply 690, and the like. It will be appreciated by those skilled in the art that the handset construction shown in fig. 15 is not limiting of the handset and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
The following describes the components of the mobile phone in detail with reference to fig. 15:
the RF circuit 610 may be configured to receive and transmit signals during a message or a call, and in particular, receive downlink information of a base station and process the downlink information with the processor 680; in addition, the data of the design uplink is sent to the base station. Typically, the RF circuitry 610 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, and the like. In addition, the RF circuitry 610 may also communicate with networks and other devices via wireless communications. The wireless communications may use any communication standard or protocol including, but not limited to, global system for mobile communications (Global System of Mobile communication, GSM), general packet radio service (General Packet Radio Service, GPRS), code division multiple access (Code Division Multiple Access, CDMA), wideband code division multiple access (Wideband Code Division Multiple Access, WCDMA), long term evolution (Long Term Evolution, LTE), email, short message service (Short Messaging Service, SMS), and the like.
The memory 620 may be used to store software programs and modules, and the processor 680 may perform various functional applications and data processing of the cellular phone by executing the software programs and modules stored in the memory 620. The memory 620 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, application programs required for at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.
The input unit 630 may be used to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the handset. In particular, the input unit 630 may include a touch panel 631 and other input devices 632. The touch panel 631, also referred to as a touch screen, may collect touch operations thereon or thereabout by a user (e.g., operations of the user on the touch panel 631 or thereabout using any suitable object or accessory such as a finger, a stylus, etc.), and drive the corresponding connection device according to a preset program. Alternatively, the touch panel 631 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch azimuth of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 680 and can receive commands from the processor 680 and execute them. In addition, the touch panel 631 may be implemented in various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 630 may include other input devices 632 in addition to the touch panel 631. In particular, other input devices 632 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, mouse, joystick, etc.
The display unit 640 may be used to display information input by a user or information provided to the user and various menus of the mobile phone. The display unit 640 may include a display panel 641, and optionally, the display panel 641 may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 631 may cover the display panel 641, and when the touch panel 631 detects a touch operation thereon or thereabout, the touch panel 631 is transferred to the processor 680 to determine the type of the touch event, and then the processor 680 provides a corresponding visual output on the display panel 641 according to the type of the touch event. Although in fig. 15, the touch panel 631 and the display panel 641 are two independent components to implement the input and input functions of the mobile phone, in some embodiments, the touch panel 631 and the display panel 641 may be integrated to implement the input and output functions of the mobile phone.
The handset may also include at least one sensor 650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel 641 according to the brightness of ambient light, and a proximity sensor that may turn off the display panel 641 and/or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the acceleration in all directions (generally three axes), and can detect the gravity and direction when stationary, and can be used for applications of recognizing the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and knocking), and the like; other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that may also be configured with the handset are not described in detail herein.
Audio circuitry 660, speaker 661, microphone 662 may provide an audio interface between a user and the handset. The audio circuit 660 may transmit the received electrical signal converted from audio data to the speaker 661, and the electrical signal is converted into a sound signal by the speaker 661 to be output; on the other hand, microphone 662 converts the collected sound signals into electrical signals, which are received by audio circuit 660 and converted into audio data, which are processed by audio data output processor 680 for transmission to, for example, another cell phone via RF circuit 610, or which are output to memory 620 for further processing.
WiFi belongs to a short-distance wireless transmission technology, and a mobile phone can help a user to send and receive emails, browse webpages, access streaming media and the like through a WiFi module 670, so that wireless broadband Internet access is provided for the user. Although fig. 15 shows a WiFi module 670, it is understood that it does not belong to the necessary constitution of the mobile phone, and can be omitted entirely as required within the scope of not changing the essence of the invention.
Processor 680 is a control center of the handset, connects various parts of the entire handset using various interfaces and lines, performs various functions of the handset and processes data by running or executing software programs and/or modules stored in memory 620, and invoking data stored in memory 620. Optionally, processor 680 may include one or more processing units; alternatively, processor 680 may integrate an application processor that primarily handles operating systems, user interfaces, applications, etc., with a modem processor that primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 680.
The handset further includes a power supply 690 (e.g., a battery) for powering the various components, optionally in logical communication with the processor 680 through a power management system, thereby implementing functions such as charge, discharge, and power management through the power management system.
Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which will not be described herein.
The steps performed by the terminal device in the above-described embodiments may be based on the terminal device structure shown in fig. 15.
Embodiments of the present application also provide a computer-readable storage medium having a computer program stored therein, which when run on a computer, causes the computer to perform the method as described in the foregoing embodiments.
Embodiments of the present application also provide a computer program product comprising a program which, when run on a computer, causes the computer to perform the method described in the previous embodiments.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (20)
1. The texture and normal library construction method is characterized by comprising the following steps:
acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map has K face areas, N is an integer greater than 1, and K is an integer greater than 1;
generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
determining a first principal component analysis parameter according to the first sampling face texture map set aiming at a target face region, wherein the target face region belongs to any face region in the K face regions;
for the target face region, if the face texture and normal library construction condition is met, the first principal component analysis parameter is used as a texture normal basis corresponding to the target face region in the face texture and normal library, wherein the face texture and normal library also comprises a texture normal basis corresponding to the K-1 face region;
The determining, for the target face area, a first principal component analysis parameter according to the first sampled face texture map set includes: aiming at the target face area, acquiring a first low average value, a first low principal component coefficient and a first low principal component standard deviation according to the first sampling face texture atlas; aiming at the target face area, acquiring a first high average value according to the first original face texture atlas; acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with the first original face texture map; for the target face region, determining a first high principal component according to the first low average value, the first low principal component coefficient, the first low principal component standard deviation, the first high average value and the first texture fitting coefficient; for the target face region, acquiring the first principal component analysis parameters according to the first high average value, the first low average value, the first high principal component and the first low principal component;
Or,
after the first original face texture map set is obtained, a first high mean value, a first high principal component coefficient and a first high principal component standard deviation are obtained according to the first original face texture map set aiming at the target face region; for the target face region, determining a first high principal component according to the first high principal component coefficient and the first high principal component standard deviation; the determining, for the target face area, a first principal component analysis parameter according to the first sampled face texture map set includes: aiming at the target face area, acquiring a first low average value, a first low principal component coefficient and a first low principal component standard deviation according to the first sampling face texture atlas; determining a first low principal component according to the first low principal component coefficient and the first low principal component standard deviation for the target face region; and aiming at the target face area, acquiring the first principal component analysis parameters according to the first high average value, the first low average value, the first high principal component and the first low principal component.
2. The texture and normal library construction method according to claim 1, wherein the obtaining a first original face texture map set includes:
Acquiring a second original face texture map set, wherein the second original face texture map set comprises M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
and carrying out disturbance processing on any one of the second original face texture maps in the second original face texture map set to obtain at least one first original face texture map contained in the first original face texture map set, wherein the disturbance processing comprises at least one of rotation, translation and scaling.
3. The texture and normal library construction method according to claim 2, wherein the obtaining the second original face texture map set includes:
obtaining M face texture maps, wherein each face texture map comprises data of a red channel, data of a green channel and data of a blue channel;
obtaining M face normal maps, wherein each face normal map comprises data of a red channel, data of a green channel and data of a blue channel, and the face normal maps and the face texture maps have a corresponding relation;
And combining each face texture map with the corresponding face normal map to obtain the second original face texture map set, wherein each second original face texture map comprises data of two red channels, data of two green channels and data of two blue channels.
4. The texture and normal library construction method according to claim 1, wherein the obtaining, for the target face region, a first low average value, a first low principal component coefficient, and a first low principal component standard deviation from the first sampled face texture atlas includes:
for the target face area, performing principal component analysis on the first sampling face texture atlas to obtain the first low-average value and a low principal component coefficient to be processed, wherein the low principal component coefficient to be processed comprises Q feature vectors, and Q is an integer greater than 1;
aiming at the target face area, acquiring a low principal component variance to be processed according to the low principal component coefficient to be processed, wherein the low principal component variance to be processed comprises Q characteristic values, and the characteristic values have a corresponding relation with the characteristic vectors;
Aiming at the target face area, arranging Q characteristic values included in the low principal component variance to be processed according to the sequence from big to small to obtain Q characteristic values after sequencing;
for the target face region, acquiring T characteristic values with the characteristic value duty ratio larger than a duty ratio threshold value from the Q characteristic values after sequencing, wherein T is an integer larger than or equal to 1 and smaller than or equal to Q;
aiming at the target face area, acquiring the first low principal component standard deviation according to the T characteristic values;
for the target face region, determining T feature vectors corresponding to the T feature values from Q feature vectors included in the low principal component coefficients to be processed, and acquiring the first low principal component coefficients according to the T feature vectors.
5. The texture and normal library construction method is characterized by comprising the following steps:
acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map has K face areas, N is an integer greater than 1, and K is an integer greater than 1;
Generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
determining a first principal component analysis parameter according to the first sampling face texture map set aiming at a target face region, wherein the target face region belongs to any face region in the K face regions;
acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with the first original face texture map;
for the target face area, determining N first loss values according to the first texture fitting coefficient, the first original face texture map set and a first high mean value and a first high principal component included in the first principal component analysis parameter, wherein the first loss values and the first original face texture map have a corresponding relation;
For the target face region, if the difference value between the maximum loss value in the N first loss values and the maximum loss value in the M second loss values is smaller than or equal to a difference threshold value, determining that the face texture and normal library construction condition is met, wherein the M second loss values are determined according to a second original face texture atlas, and the second original face texture atlas generates the first original face texture atlas after disturbance processing;
and aiming at the target face region, if the face texture and normal library construction condition is met, taking the first principal component analysis parameter as a texture normal basis corresponding to the target face region in the face texture and normal library, wherein the face texture and normal library also comprises a texture normal basis corresponding to the K-1 face region.
6. The texture and normals library construction method according to claim 5, wherein prior to the obtaining the first set of original face texture maps, the method further comprises:
acquiring a second original face texture map set, wherein the second original face texture map set comprises M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
Generating a second sampling face texture map set according to the second original face texture map set, wherein the second sampling face texture map set comprises M second sampling face texture maps, each second sampling face texture map is obtained by downsampling the second original face texture map, and each second sampling face texture map is provided with K face areas;
determining a second principal component analysis parameter according to the second sampling face texture map set aiming at the target face region, wherein the second principal component analysis parameter comprises a second high average value, a second low average value, a second high principal component and a second low principal component;
obtaining a second texture fitting coefficient, wherein the second texture fitting coefficient comprises M texture vectors, and each texture vector in the M texture vectors has a corresponding relation with the second original face texture map;
and aiming at the target face area, determining the M second loss values according to the second texture fitting coefficient, the second original face texture map set and a second high average value and a second high principal component included by the second principal component analysis parameter, wherein the second loss values have a corresponding relation with the second original face texture map.
7. The texture and normative library construction method according to claim 5, further comprising:
aiming at the target face region, if the difference value between the maximum loss value in the N first loss values and the maximum loss value in the M second loss values is larger than the difference value threshold, arranging the N first loss values in the order from large to small to obtain N first loss values after sequencing;
for the target face area, acquiring the first loss values of the first N number of the first loss values which are arranged in front from the first loss values of the N number of the first loss values which are arranged in sequence, wherein H is an integer which is greater than or equal to 1 and less than or equal to N;
determining corresponding H first original face texture maps according to the H first loss values;
and carrying out disturbance processing on the H first original face texture maps to obtain at least one third original face texture map, wherein the disturbance processing comprises at least one of rotation, translation and scale scaling.
8. A texture and normal map generation method, comprising:
acquiring an initial face texture map, wherein the initial face texture map corresponds to a first size;
Determining a K face area according to the initial face texture map, wherein K is an integer greater than 1;
for each face region included in the initial face texture map, acquiring target region maps respectively corresponding to the K face regions through a face texture and normal library, wherein the face texture and normal library comprises K groups of texture normal bases, each group of texture normal bases corresponds to one face region, and the face texture and normal library is obtained by adopting the texture and normal library construction method according to any one of claims 1 to 7;
and generating a target face texture map according to the target region maps respectively corresponding to the K face regions, wherein the target face texture map corresponds to a second size, and the second size is larger than the first size.
9. The texture and normal map generation method according to claim 8, further comprising:
acquiring an image selection instruction through a terminal device, wherein the image selection instruction is used for indicating a target conversion image;
acquiring the target conversion image from an image set to be converted according to the image selection instruction, wherein the image set to be converted comprises at least one image to be converted;
The obtaining the initial face texture map includes:
acquiring the initial face texture map obtained through shooting through terminal equipment;
after generating the target face texture map according to the target region maps respectively corresponding to the K face regions, the method further comprises:
generating a synthetic face image according to the target face texture map and the target conversion image;
and displaying the synthesized face image through the terminal equipment.
10. A texture and normal library construction apparatus comprising:
the acquisition module is used for acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map is provided with K face areas, N is an integer greater than 1, and K is an integer greater than 1;
the generating module is used for generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
The determining module is used for determining a first principal component analysis parameter according to the first sampling face texture map set aiming at a target face area, wherein the target face area belongs to any face area in the K face areas;
the construction module is used for aiming at the target face area, if the construction conditions of the face texture and the normal library are met, the first principal component analysis parameter is used as a texture normal basis corresponding to the target face area in the face texture and normal library, wherein the face texture and normal library also comprises a texture normal basis corresponding to the K-1 face area;
the determining module is specifically configured to: aiming at the target face area, acquiring a first low average value, a first low principal component coefficient and a first low principal component standard deviation according to the first sampling face texture atlas; aiming at the target face area, acquiring a first high average value according to the first original face texture atlas; acquiring a first texture fitting coefficient, wherein the first texture fitting coefficient comprises N texture vectors, and each texture vector in the N texture vectors has a corresponding relation with the first original face texture map; for the target face region, determining a first high principal component according to the first low average value, the first low principal component coefficient, the first low principal component standard deviation, the first high average value and the first texture fitting coefficient; for the target face region, acquiring the first principal component analysis parameters according to the first high average value, the first low average value, the first high principal component and the first low principal component;
Or,
the acquiring module is further configured to acquire, for the target face area, a first high mean value, a first high principal component coefficient, and a first high principal component standard deviation according to the first original face texture map set after the first original face texture map set is acquired;
the determining module is further configured to determine, for the target face area, a first high principal component according to the first high principal component coefficient and the first high principal component standard deviation;
the determining module is specifically configured to obtain, for the target face area, a first low average value, a first low principal component coefficient, and a first low principal component standard deviation according to the first sampled face texture atlas; determining a first low principal component according to the first low principal component coefficient and the first low principal component standard deviation for the target face region; and aiming at the target face area, acquiring the first principal component analysis parameters according to the first high average value, the first low average value, the first high principal component and the first low principal component.
11. The texture and normal library construction device according to claim 10, wherein the obtaining module is specifically configured to:
Acquiring a second original face texture map set, wherein the second original face texture map set comprises M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
and carrying out disturbance processing on any one of the second original face texture maps in the second original face texture map set to obtain at least one first original face texture map contained in the first original face texture map set, wherein the disturbance processing comprises at least one of rotation, translation and scaling.
12. The texture and normal library construction device according to claim 11, wherein the obtaining module is specifically configured to:
obtaining M face texture maps, wherein each face texture map comprises data of a red channel, data of a green channel and data of a blue channel;
obtaining M face normal maps, wherein each face normal map comprises data of a red channel, data of a green channel and data of a blue channel, and the face normal maps and the face texture maps have a corresponding relation;
And combining each face texture map with the corresponding face normal map to obtain the second original face texture map set, wherein each second original face texture map comprises data of two red channels, data of two green channels and data of two blue channels.
13. The texture and normal library construction device according to claim 10, wherein the determining module is specifically configured to:
for the target face area, performing principal component analysis on the first sampling face texture atlas to obtain the first low-average value and a low principal component coefficient to be processed, wherein the low principal component coefficient to be processed comprises Q feature vectors, and Q is an integer greater than 1;
aiming at the target face area, acquiring a low principal component variance to be processed according to the low principal component coefficient to be processed, wherein the low principal component variance to be processed comprises Q characteristic values, and the characteristic values have a corresponding relation with the characteristic vectors;
aiming at the target face area, arranging Q characteristic values included in the low principal component variance to be processed according to the sequence from big to small to obtain Q characteristic values after sequencing;
For the target face region, acquiring T characteristic values with the characteristic value duty ratio larger than a duty ratio threshold value from the Q characteristic values after sequencing, wherein T is an integer larger than or equal to 1 and smaller than or equal to Q;
aiming at the target face area, acquiring the first low principal component standard deviation according to the T characteristic values;
for the target face region, determining T feature vectors corresponding to the T feature values from Q feature vectors included in the low principal component coefficients to be processed, and acquiring the first low principal component coefficients according to the T feature vectors.
14. A texture and normal library construction apparatus comprising:
the acquisition module is used for acquiring a first original face texture map set, wherein the first original face texture map set comprises N first original face texture maps, each first original face texture map is provided with K face areas, N is an integer greater than 1, and K is an integer greater than 1;
the generating module is used for generating a first sampling face texture map set according to the first original face texture map set, wherein the first sampling face texture map set comprises N first sampling face texture maps, each first sampling face texture map is obtained by downsampling the first original face texture map, and each first sampling face texture map is provided with K face areas;
The determining module is used for determining a first principal component analysis parameter according to the first sampling face texture map set aiming at a target face area, wherein the target face area belongs to any face area in the K face areas;
the obtaining module is further configured to obtain a first texture fitting coefficient, where the first texture fitting coefficient includes N texture vectors, and each texture vector in the N texture vectors has a corresponding relationship with the first original face texture map;
the determining module is further configured to determine, for the target face area, N first loss values according to the first texture fitting coefficient, the first original face texture map set, and a first high average value and a first high principal component included in the first principal component analysis parameter, where the first loss values have a corresponding relationship with the first original face texture map;
the determining module is further configured to determine that the face texture and normal library construction condition is met if a difference value between a maximum loss value of the N first loss values and a maximum loss value of M second loss values is smaller than or equal to a difference threshold for the target face region, where the M second loss values are determined according to a second original face texture atlas, and the second original face texture atlas generates the first original face texture atlas after disturbance processing;
The construction module is used for aiming at the target face area, if the face texture and normal library construction condition is met, the first principal component analysis parameter is used as a texture normal basis corresponding to the target face area in the face texture and normal library, wherein the face texture and normal library also comprises a texture normal basis corresponding to the K-1 face area.
15. The texture and normals library construction apparatus according to claim 14 wherein,
the acquiring module is further configured to acquire a second original face texture map set before the acquiring the first original face texture map set, where the second original face texture map set includes M second original face texture maps, each second original face texture map has K face areas, and M is an integer greater than 1;
the generating module is further configured to generate a second sampled face texture map set according to the second original face texture map set, where the second sampled face texture map set includes M second sampled face texture maps, each second sampled face texture map is obtained by downsampling the second original face texture map, and each second sampled face texture map has K face areas;
The determining module is further configured to determine, for a target face area, a second principal component analysis parameter according to the second sampled face texture map set, where the second principal component analysis parameter includes a second high average value, a second low average value, a second high principal component, and a second low principal component;
the obtaining module is further configured to obtain a second texture fitting coefficient, where the second texture fitting coefficient includes M texture vectors, and each texture vector of the M texture vectors has a corresponding relationship with the second original face texture map;
the determining module is further configured to determine, for the target face area, the M second loss values according to the second texture fitting coefficient, the second set of original face texture maps, and a second high average value and a second high principal component included in the second principal component analysis parameter, where the second loss values have a correspondence with the second original face texture maps.
16. The texture and normals library construction device according to claim 14, further comprising: the system also comprises a sequencing module and a disturbance module;
The sorting module is configured to, for the target face area, sort the N first loss values according to a sequence from large to small if a difference value between a maximum loss value of the N first loss values and a maximum loss value of the M second loss values is greater than the difference threshold, so as to obtain sorted N first loss values;
the acquisition module is further configured to acquire, for the target face area, H first loss values arranged in front from the N first loss values after sorting, where H is an integer greater than or equal to 1 and less than or equal to the N;
the determining module is further configured to determine corresponding H first original face texture maps according to the H first loss values;
the disturbance module is configured to perform disturbance processing on the H first original face texture maps to obtain at least one third original face texture map, where the disturbance processing includes at least one of rotation, translation, and scaling.
17. A texture and normal map generation apparatus, comprising:
the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring an initial face texture map, and the initial face texture map corresponds to a first size;
The determining module is used for determining K face areas according to the initial face texture map, wherein K is an integer greater than 1;
the obtaining module is further configured to obtain, for each face region included in the initial face texture map, a target region map corresponding to each of the K face regions through a face texture and normal library, where the face texture and normal library includes K groups of texture normal bases, each group of texture normal bases corresponds to a face region, and the face texture and normal library is obtained by using the texture and normal library construction method according to any one of claims 1 to 7;
and the generating module is used for generating a target face texture map according to the target region maps respectively corresponding to the K face regions, wherein the target face texture map corresponds to a second size, and the second size is larger than the first size.
18. The texture and normal map generation apparatus according to claim 17, further comprising: a display module;
the acquisition module is further used for acquiring an image selection instruction through the terminal equipment, wherein the image selection instruction is used for indicating a target conversion image;
The acquisition module is further configured to acquire the target conversion image from a to-be-converted image set according to the image selection instruction, where the to-be-converted image set includes at least one to-be-converted image;
the acquisition module is specifically used for acquiring the initial face texture map obtained through shooting through terminal equipment;
the generating module is further configured to generate a synthetic face image according to the target face texture map and the target conversion image after generating the target face texture map according to the target region maps corresponding to the K face regions respectively;
and the display module is used for displaying the synthesized face image through the terminal equipment.
19. A computer device, comprising: memory, transceiver, processor, and bus system;
wherein the memory is used for storing programs;
the processor is configured to execute a program in the memory, and the processor is configured to execute the texture and normal library construction method according to any one of claims 1 to 7 or the texture and normal map generation method according to any one of claims 8 to 9 according to instructions in the program code;
The bus system is used for connecting the memory and the processor so as to enable the memory and the processor to communicate.
20. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the texture and normal library construction method of any one of claims 1 to 7 or to perform the texture and normal map generation method of any one of claims 8 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010837186.1A CN111914106B (en) | 2020-08-19 | 2020-08-19 | Texture and normal library construction method, texture and normal map generation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010837186.1A CN111914106B (en) | 2020-08-19 | 2020-08-19 | Texture and normal library construction method, texture and normal map generation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111914106A CN111914106A (en) | 2020-11-10 |
| CN111914106B true CN111914106B (en) | 2023-10-13 |
Family
ID=73278301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010837186.1A Active CN111914106B (en) | 2020-08-19 | 2020-08-19 | Texture and normal library construction method, texture and normal map generation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111914106B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177879A (en) * | 2021-04-30 | 2021-07-27 | 北京百度网讯科技有限公司 | Image processing method, image processing apparatus, electronic device, and storage medium |
| CN113610971B (en) * | 2021-09-13 | 2024-07-02 | 杭州海康威视数字技术股份有限公司 | Fine-grained three-dimensional model construction method and device and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916454A (en) * | 2010-04-08 | 2010-12-15 | 董洪伟 | Method for reconstructing high-resolution human face based on grid deformation and continuous optimization |
| WO2016029768A1 (en) * | 2014-08-29 | 2016-03-03 | 厦门幻世网络科技有限公司 | 3d human face reconstruction method and apparatus |
-
2020
- 2020-08-19 CN CN202010837186.1A patent/CN111914106B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101916454A (en) * | 2010-04-08 | 2010-12-15 | 董洪伟 | Method for reconstructing high-resolution human face based on grid deformation and continuous optimization |
| WO2016029768A1 (en) * | 2014-08-29 | 2016-03-03 | 厦门幻世网络科技有限公司 | 3d human face reconstruction method and apparatus |
Non-Patent Citations (1)
| Title |
|---|
| 基于形变模型的三维人脸识别;牛菓;《万方数据库》;摘要，第11-19页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111914106A (en) | 2020-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111652121B (en) | Training method of expression migration model, and method and device for expression migration | |
| WO2020253663A1 (en) | Artificial intelligence-based image region recognition method and apparatus, and model training method and apparatus | |
| US20220261960A1 (en) | Super-resolution reconstruction method and related apparatus | |
| CN111556278B (en) | Video processing method, video display device and storage medium | |
| US20210152751A1 (en) | Model training method, media information synthesis method, and related apparatuses | |
| JP2022527155A (en) | Animation character driving method and related equipment based on artificial intelligence | |
| CN110852942B (en) | Model training method, and media information synthesis method and device | |
| CN110517339B (en) | Animation image driving method and device based on artificial intelligence | |
| CN112990390B (en) | Training method of image recognition model, and image recognition method and device | |
| CN111209423B (en) | Image management method and device based on electronic album and storage medium | |
| CN112562019A (en) | Image color adjusting method and device, computer readable medium and electronic equipment | |
| CN110738211A (en) | object detection method, related device and equipment | |
| CN110517340B (en) | Artificial intelligence-based face model determination method and device | |
| CN111009031B (en) | Face model generation method, model generation method and device | |
| CN111709398A (en) | Image recognition method, and training method and device of image recognition model | |
| CN111914106B (en) | Texture and normal library construction method, texture and normal map generation method and device | |
| CN111556337B (en) | Media content implantation method, model training method and related device | |
| CN114418069A (en) | Method and device for training encoder and storage medium | |
| CN112203115A (en) | Video identification method and related device | |
| CN112950640A (en) | Video portrait segmentation method and device, electronic equipment and storage medium | |
| CN113822427A (en) | Model training method, image matching device and storage medium | |
| CN114722937A (en) | Abnormal data detection method and device, electronic equipment and storage medium | |
| CN110991325A (en) | Model training method, image recognition method and related device | |
| CN115170400A (en) | Video repair method, related device, equipment and storage medium | |
| CN113723168A (en) | Artificial intelligence-based subject identification method, related device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |