CN114663570A - Map generation method and device, electronic device and readable storage medium - Google Patents
Map generation method and device, electronic device and readable storage medium Download PDFInfo
- Publication number
- CN114663570A CN114663570A CN202210308961.3A CN202210308961A CN114663570A CN 114663570 A CN114663570 A CN 114663570A CN 202210308961 A CN202210308961 A CN 202210308961A CN 114663570 A CN114663570 A CN 114663570A
- Authority
- CN
- China
- Prior art keywords
- face image
- illumination
- balance model
- map
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/506—Illumination models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2024—Style variation
Abstract
The invention relates to a method and a device for generating a map, an electronic device and a readable storage medium, wherein the method comprises the following steps: generating a second face image with unbalanced illumination based on the first face image with balanced illumination, and training the initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model; receiving a third face image and generating a third texture map corresponding to the third face image; acquiring a trained illumination balance model, and taking a third face image and a third texture map as the input of the trained illumination balance model; and operating the trained illumination balance model to obtain the target texture mapping with balanced illumination. And performing illumination correction on the third texture mapping through the trained illumination balance model to obtain a target texture mapping with balanced illumination, so that the influence of ambient illumination on the face image is reduced, and a good virtual image can be generated based on the target texture mapping.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a method and an apparatus for generating a map, an electronic apparatus, and a readable storage medium.
Background
With the development of technologies such as artificial intelligence, VR (Virtual Reality), AR (Augmented Reality), and the like, more and more application software supports users to create personalized Virtual images, and brings more diversified and more immersive entertainment and social ways to users. One of the indispensable techniques is three-dimensional head reconstruction; in the three-dimensional head reconstruction, a texture map corresponding to a face of a user is generated, and a virtual image corresponding to the user is obtained according to the texture map; however, the texture map obtained in this way is easily affected by the ambient lighting, and the generated texture usually has uneven lighting such as shadow and highlight, so that the virtual image is abnormally displayed.
Disclosure of Invention
The invention provides a map generating method, a map generating device, an electronic device and a readable storage medium, and aims to solve the technical problem that in the prior art, due to the influence of ambient illumination on a texture map, an avatar is abnormally displayed.
In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides a map generating method, comprising:
generating a second face image with unbalanced illumination based on the first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model;
receiving a third face image and generating a third texture map corresponding to the third face image;
acquiring a trained illumination balance model, and taking the third face image and the third texture map as the input of the trained illumination balance model;
and operating the trained illumination balance model to obtain the target texture mapping with balanced illumination.
Optionally, the step of generating a second face image with unbalanced illumination based on the first face image with balanced illumination, and training the initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model includes:
acquiring a first face image with balanced illumination from a database, and performing relighting operation on the first face image to obtain a second face image with unbalanced illumination;
generating a first texture mapping corresponding to the first face image and a second texture mapping corresponding to the second face image, and constructing a training sample by adopting the first texture mapping, the second face image and the second texture mapping;
and acquiring an initial illumination balance model, and training the initial illumination balance model through the training sample to obtain a trained illumination balance model.
Optionally, the step of obtaining the illumination balanced first face image from the database includes:
calculating a brightness mean value of the fourth face image for each fourth face image in a database;
judging whether the brightness mean value is larger than a preset brightness threshold value or not;
and if the brightness mean value is larger than a preset brightness threshold value, taking the fourth face image corresponding to the brightness mean value as the first face image.
Optionally, the step of calculating the brightness mean value of the fourth face image includes:
respectively calculating the brightness value corresponding to each pixel point in the fourth face image;
and calculating the average value of each brightness value to obtain the brightness average value.
Optionally, the step of calculating the brightness mean value of the fourth face image includes:
respectively calculating the brightness value corresponding to each pixel point in the fourth face image;
and calculating the standard deviation of each brightness value to obtain the brightness mean value.
Optionally, the step of training the initial illumination equalization model by the training sample includes:
taking the second face image and the second texture mapping as the input of the initial illumination balance model, and obtaining a fourth texture mapping through the initial illumination balance model;
acquiring a preset loss function, and substituting the first texture map and the fourth texture map into the preset loss function to calculate the confrontation loss;
optimizing the initial illumination equalization model by the countervailing loss.
Optionally, the step of generating a third texture map corresponding to the third face image includes:
carrying out reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;
and performing two-dimensional mapping on pixel points in the first three-dimensional face model to obtain the third texture map.
In order to achieve the above object, the present invention further provides a map generating apparatus, including:
the first training module is used for generating a second face image with unbalanced illumination based on a first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model;
the first generation module is used for receiving a third face image and generating a third texture map corresponding to the third face image;
the first acquisition module is used for acquiring the trained illumination balance model and taking the third face image and the third texture map as the input of the trained illumination balance model;
and the first execution module is used for operating the trained illumination balance model to obtain a target texture mapping with balanced illumination.
Optionally, the first training module comprises:
the first acquisition sub-module is used for acquiring a first face image with balanced illumination from a database and carrying out relighting operation on the first face image to obtain a second face image with unbalanced illumination;
the first generation submodule is used for generating a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and constructing a training sample by adopting the first texture map, the second face image and the second texture map;
and the first training submodule is used for acquiring an initial illumination balance model and training the initial illumination balance model through the training sample to obtain a trained illumination balance model.
Optionally, the first obtaining sub-module includes:
the first calculation unit is used for calculating the brightness mean value of each fourth face image in the database;
the first judgment unit is used for judging whether the brightness mean value is larger than a preset brightness threshold value or not;
and the first execution unit is used for taking the fourth face image corresponding to the brightness mean value as the first face image if the brightness mean value is greater than a preset brightness threshold value.
Optionally, the first computing unit includes:
the first calculating subunit is configured to calculate, respectively, a luminance value corresponding to each pixel point in the fourth face image;
and the second calculating subunit is used for calculating the average value of each brightness value to obtain the brightness average value.
Optionally, the first computing unit includes:
a fourth calculating subunit, configured to calculate, respectively, luminance values corresponding to the pixel points in the fourth face image;
and the fifth calculating subunit calculates the standard deviation of each brightness value to obtain the brightness mean value.
Optionally, the first training submodule comprises:
the second execution unit is used for taking the second face image and the second texture mapping as the input of the initial illumination balance model and obtaining a fourth texture mapping through the initial illumination balance model;
the second calculation unit is used for obtaining a preset loss function and substituting the first texture mapping and the fourth texture mapping into the preset loss function to calculate the confrontation loss;
a second execution unit, configured to optimize the initial illumination balancing model through the countervailing loss.
Optionally, the first generating module comprises:
the first execution sub-module is used for carrying out reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;
and the second execution submodule is used for carrying out two-dimensional mapping on pixel points in the first three-dimensional face model to obtain the third texture map.
To achieve the above object, the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the map generation method as described above.
To achieve the above object, the present invention further provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the steps of the map generation method as described above.
According to the generating method and device of the map, the electronic device and the readable storage medium, a second face image with unbalanced illumination is generated based on a first face image with balanced illumination, and an initial illumination balance model is trained through the first face image and the second face image to obtain a trained illumination balance model; receiving a third face image and generating a third texture map corresponding to the third face image; acquiring a trained illumination balance model, and taking the third face image and the third texture map as the input of the trained illumination balance model; and operating the trained illumination balance model to obtain the target texture mapping with balanced illumination. And performing illumination correction on a third texture mapping corresponding to a third face image through the trained illumination balance model, so that a target texture mapping with balanced illumination can be obtained, the influence of environmental illumination on the face image is reduced, and a good virtual image can be generated and displayed based on the target texture mapping.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a schematic flow chart diagram illustrating a first embodiment of a map generation method according to the present invention;
FIG. 2 is a schematic flow chart illustrating model training in a second embodiment of a chartlet generation method according to the present invention;
FIG. 3 is a schematic structural diagram of an illumination balance model in the map generation method of the present invention;
FIG. 4 is a schematic view illustrating a selection process of a first face image in the generating method of a map according to the present invention;
fig. 5 is a schematic block diagram of an electronic device according to the present invention.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In order to make the technical field of the invention better understand the scheme of the invention, the invention is combined with the attached drawings in the embodiment of the invention
Technical solutions in the embodiments are clearly and completely described, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.
The invention provides a map generating method, referring to fig. 1, fig. 1 is a schematic flow chart of a first embodiment of the map generating method of the invention, and the method comprises the following steps:
step S10, generating a second face image with unbalanced illumination based on a first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model;
the illumination balance refers to that the difference between the brightness of each part of the face image is small, and whether the illumination of the face image is balanced or not can be determined according to the setting parameters of the actual application scene, such as the average value or the standard deviation of each pixel in the face image, and the specific setting mode refers to the following embodiments. The first face image is a preselected face image which meets the requirement of illumination balance; and adjusting the illumination parameters of the first face image so that the adjusted first face image does not accord with the illumination balance requirement, and obtaining a second face image. The initial illumination balance model is trained through the first face image and the second face image, so that the illumination balance model has the capability of converting the face image with unbalanced illumination into the face image with balanced illumination.
Step S20, receiving a third face image and generating a third texture map corresponding to the third face image;
the third face image can be an image selected and input by a user or an image acquired by image acquisition equipment such as a camera in real time; it can be understood that the input or collected image usually includes environmental information in addition to the face, and therefore, a face recognition operation needs to be performed on the image to extract pixels only including the face from the image, and a third face image is generated according to the pixels only including the face. The specific face recognition method may be set according to the actual application scenario and the need, and is not limited herein.
The map generation method of the embodiment is applied to the establishment of the virtual image; the virtual image is obtained by creating a character three-dimensional model and covering a map on the surface of the character three-dimensional model; because the surface of the character three-dimensional model is three-dimensional, and the third face image is planar, the third face image needs to be converted into a third texture map and then covered on the surface of the human body three-dimensional model to obtain an avatar.
The step S20 of generating a third texture map corresponding to the third face image includes the steps of:
step S21, carrying out reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;
step S22, performing two-dimensional mapping on the pixel points in the first three-dimensional face model to obtain the third texture map.
The reconstruction operation refers to an operation of converting a planar face image into a three-dimensional face model. It should be noted that specific reconstruction operations may be selected according to an actual application scenario and needs, for example, in this embodiment, a third Face image is reconstructed by using a 3D digital media Model (3D deformable Model), such as a BFM (base Face Model, three-dimensional Basel Face Model); specifically, a three-dimensional deformable model coefficient recurrent neural network is obtained, and the three-dimensional deformable model coefficient recurrent neural network can be selected from the existing public models; inputting the third face image into a three-dimensional deformable model coefficient regression neural network to obtain a predicted BFM coefficient, wherein: the BFM coefficient includes (c)i，ce，ct， p，γ)∈R257Wherein R is257Is a 257 dimensional vector, i.e. comprises 257 unit data; c. CiAs identity factor, ci∈R80Namely, the identity coefficient corresponds to an 80-dimensional vector and comprises 80 unit data; c. CeIs an expression coefficient, ce∈R64Namely 64-dimensional vectors corresponding to the expression coefficients and comprising 64 unit data; c. CtAs texture mapping coefficients, ct∈R80That is, the texture mapping coefficients correspond to 80-dimensional vectors, and comprise 80 unit data; p is the head pose coefficient, p is belonged to R6I.e. the head pose coefficients correspond to 6-dimensional vectors, including6 units of data; gamma is the illumination coefficient, gamma belongs to R27Namely, the illumination coefficient corresponds to a 27-dimensional vector and comprises 27 unit data; it should be noted that the specific BFM coefficient may be set according to the actual application model and the need, and the above description only provides an optional solution; inputting the obtained BFM coefficient into a geometric shape formula S of the BFM; the concrete description is as follows:
wherein the content of the first and second substances,
is the average shape of BFM, BidIs the identity base of BFM, BexpAn expression base of BFM;
Bidand BexpAre all preset parameters of the geometric shape formula S. And obtaining the first three-dimensional face model through a geometric shape formula.
Two-dimensional mapping refers to the operation of mapping points in a three-dimensional model to a two-dimensional plane. It should be noted that a specific mapping manner may be selected according to an actual application scenario and needs, for example, in this embodiment, the first three-dimensional face model is two-dimensionally mapped by using a Texture Sampling Texture method to obtain a third Texture map. Specifically, the formula is described as:
V2d＝Pr·(R·+t3d)
wherein: v2dMapping point coordinates on a two-dimensional plane for the first three-dimensional face model; pr is an orthogonal projection matrix, namely a camera coefficient; t is t3dAnd the head pose coefficient is obtained by p conversion. In obtaining V2dThen, the first three-dimensional face model is matched with the V2dSampling corresponding positions, and taking a set of pixels obtained by sampling as a third texture map; specifically, the formula is described as:
T＝F(V2d)
wherein T is a third texture map, and F is a sampling function; it should be noted that the sampling function may be selectively set according to an actual application scenario and needs, and is not limited herein. And obtaining a third texture map corresponding to the third face image by the steps.
Step S30, acquiring a trained illumination balance model, and taking the third face image and the third texture map as the input of the trained illumination balance model;
and step S40, operating the trained illumination balance model to obtain an illumination-balanced target texture map.
And the illumination balance model is used for carrying out balance processing on the third texture mapping so as to convert the third texture mapping with uneven illumination into a target texture mapping with even illumination. Specifically, the illumination balance model in this embodiment adopts a generation countermeasure network model. Generating a confrontation network model is a deep learning model. It should be noted that the model can also be selected based on the actual application scenario and the need.
In this embodiment, the illumination correction is performed on the third texture map corresponding to the third face image through the trained illumination balance model, so that the target texture map with balanced illumination can be obtained, the influence of ambient illumination on the face image is reduced, and a good virtual image can be generated and displayed based on the target texture map.
Further, referring to fig. 2, in the second embodiment of the map generating method according to the present invention proposed based on the first embodiment of the present invention, the step S10 includes the steps of:
step S11, acquiring a first face image with balanced illumination from a database, and performing relighting operation on the first face image to obtain a second face image with unbalanced illumination;
a plurality of face images, namely a fourth face image, are stored in the database; the fourth face image may be obtained in a variety of ways, such as being obtained from a network or a server, or being captured by an image capture device, etc. The relighting operation is a superposition operation of simulating illumination on the image, so that the image has the illumination characteristic; it should be noted that a specific Relighting operation mode may be selected according to an actual application scenario and needs, for example, a DPR (Deep Single Image portal lighting) neural network is adopted to perform Relighting operation in this embodiment; specifically, a plurality of groups of illumination parameters are preset, wherein the illumination parameters comprise different illumination parameters such as direction, brightness and the like; when the re-illumination operation is needed, a group of illumination parameters are randomly selected, the selected illumination parameters and the first face image serve as input of a DPR neural network, and the DPR neural network outputs a second face image superposed with the illumination parameters based on the illumination parameters and the first face image.
Step S12, generating a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and constructing a training sample by using the first texture map, the second face image and the second texture map;
the generation of the first texture map and the second texture map can be performed by analogy with the generation of the third texture map, which is not described herein again. It should be noted that, because the first face image and the second face image are different only in terms of illumination characteristics, the reconstruction results of the first face image and the second face image should be the same, so that only the reconstruction operation may be performed on the first face image, and then the first texture map is obtained according to the second three-dimensional face model obtained by the reconstruction operation; and simultaneously, referring the second face image to a second three-dimensional face model to obtain a second texture mapping.
It should be noted that a group of training samples includes a first texture map, a second face image, and a second texture map, the number of the training samples is multiple, and a single training sample is used for performing one-time training on the initial illumination equalization model.
And step S13, obtaining an initial illumination balance model, and training the initial illumination balance model through the training sample to obtain a trained illumination balance model.
Referring to fig. 3, the illumination equalization model in this embodiment is used for generating a confrontation network model, and the generation of the confrontation network model includes a generator and a discriminator, where the generator is composed of two encoders and a decoder, the encoders include an image encoder and a texture encoder, the image encoder is used for extracting feature information of a face image, the texture encoder is used for extracting feature information of a texture map, and the feature information extracted by the image encoder and the texture encoder is cascaded through feature channels and then input into the decoder to obtain the texture map; the generator in this embodiment adopts a U-Net structure, and the discriminator adopts a Patch-GAN structure. In the training process, the generator is used for generating a predicted texture mapping, namely a fourth texture mapping, according to the second face image with unbalanced illumination and the corresponding second texture mapping; the discriminator is used for comparing the fourth texture map with the first texture map so as to correct the deviation; and in the using process after the training is finished, inputting the third face image and the third texture mapping into the generator to obtain the target texture mapping.
In the embodiment, a first face image with balanced illumination is obtained by selection, and a second face image with unbalanced illumination is obtained by re-illumination operation; and then training the initial illumination balance model according to the first face image with balanced illumination and the second face image with unbalanced illumination to obtain a trained illumination balance model.
Further, referring to fig. 4, in a third embodiment of the method for generating a map according to the present invention based on the second embodiment of the present invention, the step S40 includes the steps of:
step S111, calculating the brightness mean value of each fourth face image in the database;
it should be noted that, the selection of the first face image may be performed in an image acquisition stage, that is, when one face image is acquired, whether the brightness mean value of the face image is greater than a preset brightness threshold is determined, and only if the brightness mean value is greater than the preset brightness threshold, the face image is stored in the database, and at this time, all the face images in the database are the first face images; the method can also be carried out in the stage of acquiring the training sample, namely various types of face images are stored in the database, and when the training sample needs to be provided, a fourth face image with the brightness mean value larger than a preset brightness threshold value is selected from the database to serve as the first face image.
The brightness mean value is used for reflecting the illumination uniformity of the face image; firstly, obtaining a brightness value corresponding to each pixel point in a fourth face image, and further carrying out mean value operation on all the brightness values to obtain a brightness mean value corresponding to the fourth face image; the mean value operation may be selected according to an actual application scenario and needs, for example, in this embodiment, the mean value and the standard deviation are used as the mean value operation, specifically, the mean value is used as an example; the formula is described as:
wherein, the Lum is the mean value of brightness; i is the serial number of the pixel point; i isi rThe color value corresponding to the red channel of the ith pixel point in the RGB color mode; i isi gThe color value is the color value corresponding to the green channel of the ith pixel point in the RGB color mode; i isi bThe color value corresponding to the ith pixel point blue channel in the RGB color mode; and M is the number of pixel points in the fourth face image.
The root part is characterized as the brightness value corresponding to the ith pixel point; other luminance value calculation formulas can be replaced according to the actual application requirements, such as:
at this time, the formula of the luminance mean value is described as:
the brightness value calculation method of a single pixel point is as described above, the average value calculation of the standard deviation can be performed by analogy with the above method, and is not described herein again, the average value is used for description in the subsequent embodiments, and the standard deviation scheme can be performed by analogy without repeated description.
Step S112, judging whether the brightness mean value is larger than a preset brightness threshold value;
step S113, if the brightness mean value is greater than a preset brightness threshold, taking the fourth face image corresponding to the brightness mean value as the first face image.
The preset brightness threshold value is a value used for representing whether the face image is balanced in illumination or not; the specific preset brightness threshold value can be set according to the actual application scene and the requirements, such as the environment and the like; in addition, for comparison, the brightness mean value is further subjected to normalization processing, the range of the brightness mean value is 0-255 according to the mean value calculation formula, the brightness mean value is divided by 255, the range of the brightness mean value is 0-1, and then judgment is carried out through a preset brightness threshold value; the preset luminance threshold is set to 0.98 as in the present embodiment; it can be understood that, the preset brightness threshold may also be set directly with 0-255 as the range of the brightness mean value.
And when the brightness mean value is larger than the preset brightness threshold value, the fourth face image is considered as the face image with balanced illumination, and at the moment, the fourth face image is taken as the first face image. And when the brightness mean value is less than or equal to the preset brightness threshold value, the fourth face image is considered as the face image with unbalanced illumination, and at the moment, the fourth face image is filtered.
The present embodiment can accurately select the illumination-equalized fourth face image as the first face image.
Further, in a fourth embodiment of the map generating method according to the present invention based on the second embodiment of the present invention, the step S13 includes the steps of:
step S131, using the second face image and the second texture map as the input of the initial illumination balance model, and obtaining a fourth texture map through the initial illumination balance model;
step S132, obtaining a preset loss function, and substituting the first texture map and the fourth texture map into the preset loss function to calculate a countermeasure loss;
and S133, optimizing the initial illumination balance model through the confrontation loss.
The fourth texture map is a predicted illumination balanced texture map of the initial illumination balanced model; the first texture map is a texture map with balanced illumination corresponding to the second texture map; that is, the first texture map has been determined to be light balanced, and therefore, it can be determined whether the fourth texture map is light balanced with respect to the first texture map.
The loss function is a function that maps the value of a random event or its associated random variable to a non-negative real number to represent the "risk" or "loss" of the random event. Specifically, the loss function can be set according to the actual application scenario and the need; in the embodiment, the initial illumination balance model is optimized in a mode of combining various loss functions; specifically, the loss functions include a pixel loss function, a perceptual loss function, a style loss function, and a countermeasure loss function; wherein:
the pixel loss function formula is described as follows:
wherein L isrec() Is a pixel loss function; t is a unit ofpredA fourth texture map; t is a unit of1A first texture map is obtained; n is the number of pixel points in the fourth texture map or the first texture map; i is the serial number of the pixel point.
Since the color values between the first texture map and the fourth texture map should be similar, the pixel-level difference between the first texture map and the fourth texture map is expressed as an L1 distance, i.e., a manhattan distance, between corresponding pixel points of the first texture map and the fourth texture map by introducing a pixel loss function.
High-frequency information of the texture map, such as image content, spatial structure and the like, cannot be generated well only through a pixel loss function; in other words, the pixel loss function cannot well measure the perceptual difference between the first texture map and the fourth texture map, so the perceptual loss function is introduced to promote the fourth texture map to have high-level semantic features similar to those of the first texture map, and the similarity of the images can be measured more robustly when the illumination balance model is trained. Extracting the convolution characteristics of each layer of the first texture map and the fourth texture map respectively through a VGG19 neural network by using a perceptual loss function, and expressing the similarity between the convolution characteristics of each layer as the L1 distance between the first texture map and the fourth texture map; the perceptual loss function formula is described as follows:
wherein L isperc() As a function of perceptual loss; phi is aiIs the i-th layer convolution characteristic of the VGG19 neural network.
In order to further ensure the style uniformity between the first texture map and the fourth texture map and avoid the existence of large style difference between the first texture map and the fourth texture map, such as color, texture and the like, a style loss function is introduced, and the style loss function is used for measuring the style difference between the first texture map and the fourth texture map and is specifically represented as the L1 distance of a gram matrix of convolution characteristics of each layer of the first texture map and the fourth texture map; the style loss function formula is described as follows:
to generate a high quality stylized texture map, the first texture map and the fourth texture map are each input to a discriminator, and the resulting penalty function is:
wherein L isgen() Generating a pair loss-tolerance function; d () is the confidence.
Wherein L isdis() To discriminate the penalty function against.
It should be noted that the above-mentioned loss function is only an illustration of a feasible scheme, and other loss functions or function combinations may also be selected to optimize the illumination balance model according to the actual application scenario and the need.
It should be noted that, when the initial illumination balance model is trained, training completion conditions, such as training times or model reliability, need to be set, and when the initial illumination balance model satisfies the training completion conditions, the initial illumination balance model is used as the illumination balance model after training. It can be understood that the trained illumination balance model can also be self-optimized through the texture mapping and the face image which are actually processed in the actual use process.
According to the embodiment, the trained illumination balance model can be obtained by reasonably setting the loss function to optimize the initial illumination balance model.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The present invention also provides a map generating apparatus for implementing the map generating method, the map generating apparatus including:
the first training module is used for generating a second face image with unbalanced illumination based on a first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model;
the first generation module is used for receiving a third face image and generating a third texture map corresponding to the third face image;
the first acquisition module is used for acquiring the trained illumination balance model and taking the third face image and the third texture map as the input of the trained illumination balance model;
and the first execution module is used for operating the trained illumination balance model to obtain a target texture mapping with balanced illumination.
The map generating device performs illumination correction on the third texture map corresponding to the third face image through the trained illumination balance model, so that a target texture map with balanced illumination can be obtained, the influence of environmental illumination on the face image is reduced, and a good virtual image can be generated based on the target texture map.
It should be noted that the first training module in this embodiment may be configured to execute step S10 in this embodiment, the first generating module in this embodiment may be configured to execute step S20 in this embodiment, the first obtaining module in this embodiment may be configured to execute step S30 in this embodiment, and the first executing module in this embodiment may be configured to execute step S40 in this embodiment.
Further, the first training module comprises:
the first acquisition sub-module is used for acquiring a first face image with balanced illumination from a database and carrying out relighting operation on the first face image to obtain a second face image with unbalanced illumination;
the first generation submodule is used for generating a first texture mapping corresponding to the first face image and a second texture mapping corresponding to the second face image, and constructing a training sample by adopting the first texture mapping, the second face image and the second texture mapping;
and the first training submodule is used for acquiring an initial illumination balance model and training the initial illumination balance model through the training sample to obtain a trained illumination balance model.
Further, the first obtaining sub-module includes:
the first calculation unit is used for calculating the brightness mean value of each fourth face image in the database;
the first judgment unit is used for judging whether the brightness mean value is larger than a preset brightness threshold value or not;
and the first execution unit is used for taking the fourth face image corresponding to the brightness mean value as the first face image if the brightness mean value is greater than a preset brightness threshold value.
Further, the first calculation unit includes:
the first calculating subunit is configured to calculate, respectively, a luminance value corresponding to each pixel point in the fourth face image;
and the second calculating subunit is used for calculating the average value of each brightness value to obtain the brightness average value.
Further, the first calculation unit includes:
a fourth calculating subunit, configured to calculate, respectively, luminance values corresponding to the pixel points in the fourth face image;
and the fifth calculating subunit calculates the standard deviation of each brightness value to obtain the brightness mean value.
Further, the first training module comprises:
the second execution unit is used for taking the second face image and the second texture mapping as the input of the initial illumination balance model and obtaining a fourth texture mapping through the initial illumination balance model;
the second calculation unit is used for obtaining a preset loss function and substituting the first texture mapping and the fourth texture mapping into the preset loss function to calculate the confrontation loss;
a second execution unit, configured to optimize the initial illumination balancing model through the countervailing loss.
Further, the first generating module comprises:
the first execution subunit is used for carrying out reconstruction operation on the third face image so as to obtain a first three-dimensional face model corresponding to the third face image;
and the second execution subunit is used for performing two-dimensional mapping on the pixel points in the first three-dimensional face model to obtain the third texture map.
It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. The modules may be implemented by software as part of the apparatus, or may be implemented by hardware, where the hardware environment includes a network environment.
Referring to fig. 5, the electronic device may include components such as a communication module 10, a memory 20, and a processor 30 in a hardware structure. In the electronic device, the processor 30 is connected to the memory 20 and the communication module 10, respectively, the memory 20 stores thereon a computer program, which is executed by the processor 30 at the same time, and when executed, implements the steps of the above-mentioned method embodiments.
The communication module 10 may be connected to an external communication device through a network. The communication module 10 may receive a request from an external communication device, and may also send the request, an instruction, and information to the external communication device, where the external communication device may be another electronic apparatus, a server, or an internet of things device, such as a television.
The memory 20 may be used to store software programs as well as various data. The memory 20 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function (such as generating a third texture map corresponding to the third face image), and the like; the storage data area may include a database, and the storage data area may store data or information created according to use of the system, or the like. Further, the memory 20 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The processor 30, which is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by operating or executing software programs and/or modules stored in the memory 20 and calling data stored in the memory 20, thereby performing overall monitoring of the electronic device. Processor 30 may include one or more processing units; alternatively, the processor 30 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 30.
Although not shown in fig. 5, the electronic device may further include a circuit control module, which is used for connecting with a power supply to ensure the normal operation of other components. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 5 does not constitute a limitation of the electronic device and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.
The invention also proposes a computer-readable storage medium on which a computer program is stored. The computer-readable storage medium may be the Memory 20 in the electronic apparatus in fig. 5, and may also be at least one of a ROM (Read-Only Memory)/RAM (Random Access Memory), a magnetic disk, and an optical disk, and the computer-readable storage medium includes instructions for enabling a terminal device (which may be a television, an automobile, a mobile phone, a computer, a server, a terminal, or a network device) having a processor to execute the method according to the embodiments of the present invention.
In the present invention, the terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance, and those skilled in the art can understand the specific meanings of the above terms in the present invention according to specific situations.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although the embodiment of the present invention has been shown and described, the scope of the present invention is not limited thereto, it should be understood that the above embodiment is illustrative and not to be construed as limiting the present invention, and that those skilled in the art can make changes, modifications and substitutions to the above embodiment within the scope of the present invention, and that these changes, modifications and substitutions should be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for generating a map, the method comprising:
generating a second face image with unbalanced illumination based on the first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model;
receiving a third face image and generating a third texture map corresponding to the third face image;
acquiring the trained illumination balance model, and taking the third face image and the third texture map as the input of the trained illumination balance model;
and operating the trained illumination balance model to obtain a target texture mapping with balanced illumination.
2. The method of generating a map of claim 1, wherein the step of generating a second face image with unbalanced illumination based on a first face image with balanced illumination and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model comprises:
acquiring a first face image with balanced illumination from a database, and performing relighting operation on the first face image to obtain a second face image with unbalanced illumination;
generating a first texture mapping corresponding to the first face image and a second texture mapping corresponding to the second face image, and constructing a training sample by adopting the first texture mapping, the second face image and the second texture mapping;
and acquiring an initial illumination balance model, and training the initial illumination balance model through the training sample to obtain a trained illumination balance model.
3. The map generation method of claim 2, wherein said step of obtaining a light-balanced first face image from a database comprises:
calculating a brightness mean value of the fourth face image for each fourth face image in a database;
judging whether the brightness mean value is larger than a preset brightness threshold value or not;
and if the brightness mean value is larger than a preset brightness threshold value, taking the fourth face image corresponding to the brightness mean value as the first face image.
4. The map generation method of claim 3, wherein said step of calculating a mean value of luminance of said fourth face image comprises:
respectively calculating the brightness value corresponding to each pixel point in the fourth face image;
and calculating the average value of all the brightness values to obtain the brightness average value.
5. The map generation method of claim 3, wherein said step of calculating a luminance mean value of said fourth face image comprises:
respectively calculating the brightness value corresponding to each pixel point in the fourth face image;
and calculating the standard deviation of each brightness value to obtain the brightness mean value.
6. The method of generating a map of claim 2, wherein said step of training said initial illumination equalization model by said training samples comprises:
taking the second face image and the second texture mapping as the input of the initial illumination balance model, and obtaining a fourth texture mapping through the initial illumination balance model;
acquiring a preset loss function, and substituting the first texture map and the fourth texture map into the preset loss function to calculate the confrontation loss;
optimizing the initial illumination equalization model by the countervailing loss.
7. The map generation method of claim 1, wherein said step of generating a third texture map corresponding to said third face image comprises:
carrying out reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;
and performing two-dimensional mapping on pixel points in the first three-dimensional face model to obtain the third texture map.
8. A map generation apparatus, characterized by comprising:
the first training module is used for generating a second face image with unbalanced illumination based on a first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model;
the first generation module is used for receiving a third face image and generating a third texture map corresponding to the third face image;
the first acquisition module is used for acquiring the trained illumination balance model and taking the third face image and the third texture map as the input of the trained illumination balance model;
and the first execution module is used for operating the trained illumination balance model to obtain a target texture mapping with balanced illumination.
9. An electronic device, characterized in that the electronic device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, implements the steps of the map generation method according to any of claims 1 to 7.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, realizes the steps of the map generation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210308961.3A CN114663570A (en) | 2022-03-25 | 2022-03-25 | Map generation method and device, electronic device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210308961.3A CN114663570A (en) | 2022-03-25 | 2022-03-25 | Map generation method and device, electronic device and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114663570A true CN114663570A (en) | 2022-06-24 |
Family
ID=82034044
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210308961.3A Pending CN114663570A (en) | 2022-03-25 | 2022-03-25 | Map generation method and device, electronic device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114663570A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024001363A1 (en) * | 2022-06-30 | 2024-01-04 | 魔门塔(苏州)科技有限公司 | Image processing method and apparatus, and electronic device |
-
2022
- 2022-03-25 CN CN202210308961.3A patent/CN114663570A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024001363A1 (en) * | 2022-06-30 | 2024-01-04 | 魔门塔(苏州)科技有限公司 | Image processing method and apparatus, and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109961507B (en) | Face image generation method, device, equipment and storage medium | |
| CN107993216B (en) | Image fusion method and equipment, storage medium and terminal thereof | |
| CN110599395B (en) | Target image generation method, device, server and storage medium | |
| CN106682632B (en) | Method and device for processing face image | |
| CN110069974B (en) | Highlight image processing method and device and electronic equipment | |
| CN112037320A (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN111882627A (en) | Image processing method, video processing method, device, equipment and storage medium | |
| US11576478B2 (en) | Method for simulating the rendering of a make-up product on a body area | |
| CN112164005A (en) | Image color correction method, device, equipment and storage medium | |
| CN110852940A (en) | Image processing method and related equipment | |
| CN113808277B (en) | Image processing method and related device | |
| CN112241933A (en) | Face image processing method and device, storage medium and electronic equipment | |
| CN110335330A (en) | Image simulation generation method and its system, deep learning algorithm training method and electronic equipment | |
| US20230074060A1 (en) | Artificial-intelligence-based image processing method and apparatus, electronic device, computer-readable storage medium, and computer program product | |
| CN115984447A (en) | Image rendering method, device, equipment and medium | |
| CN114663570A (en) | Map generation method and device, electronic device and readable storage medium | |
| CN116740261A (en) | Image reconstruction method and device and training method and device of image reconstruction model | |
| CN113421312A (en) | Method and device for coloring black and white video, storage medium and terminal | |
| CN117061882A (en) | Video image processing method, apparatus, device, storage medium, and program product | |
| CN112489144A (en) | Image processing method, image processing apparatus, terminal device, and storage medium | |
| CN112967193A (en) | Image calibration method and device, computer readable medium and electronic equipment | |
| CN113706400A (en) | Image correction method, image correction device, microscope image correction method, and electronic apparatus | |
| CN109658360B (en) | Image processing method and device, electronic equipment and computer storage medium | |
| CN113724282A (en) | Image processing method and related product | |
| CN111539353A (en) | Image scene recognition method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |